2023-04-05 18:53:30,913 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a 2023-04-05 18:53:30,926 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-05 18:53:30,943 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-05 18:53:30,943 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/cluster_e107b28c-27c6-6ca4-18a0-71d30edffa22, deleteOnExit=true 2023-04-05 18:53:30,944 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-05 18:53:30,945 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/test.cache.data in system properties and HBase conf 2023-04-05 18:53:30,945 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/hadoop.tmp.dir in system properties and HBase conf 2023-04-05 18:53:30,946 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/hadoop.log.dir in system properties and HBase conf 2023-04-05 18:53:30,946 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-05 18:53:30,947 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-05 18:53:30,947 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-05 18:53:31,066 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-05 18:53:31,490 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-05 18:53:31,495 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-05 18:53:31,495 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-05 18:53:31,495 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-05 18:53:31,495 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-05 18:53:31,496 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-05 18:53:31,497 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-05 18:53:31,497 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-05 18:53:31,497 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-05 18:53:31,498 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-05 18:53:31,498 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/nfs.dump.dir in system properties and HBase conf 2023-04-05 18:53:31,499 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/java.io.tmpdir in system properties and HBase conf 2023-04-05 18:53:31,499 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-05 18:53:31,499 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-05 18:53:31,499 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-05 18:53:32,103 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-05 18:53:32,108 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-05 18:53:32,411 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-05 18:53:32,594 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-05 18:53:32,621 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-05 18:53:32,665 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-05 18:53:32,696 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/java.io.tmpdir/Jetty_localhost_33717_hdfs____.9k10k9/webapp 2023-04-05 18:53:32,849 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33717 2023-04-05 18:53:32,897 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-05 18:53:32,897 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-05 18:53:33,361 WARN [Listener at localhost/38439] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-05 18:53:33,509 WARN [Listener at localhost/38439] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-05 18:53:33,528 WARN [Listener at localhost/38439] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-05 18:53:33,534 INFO [Listener at localhost/38439] log.Slf4jLog(67): jetty-6.1.26 2023-04-05 18:53:33,540 INFO [Listener at localhost/38439] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/java.io.tmpdir/Jetty_localhost_44855_datanode____.tazao1/webapp 2023-04-05 18:53:33,640 INFO [Listener at localhost/38439] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44855 2023-04-05 18:53:33,984 WARN [Listener at localhost/43705] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-05 18:53:34,508 DEBUG [Listener at localhost/43705] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a 2023-04-05 18:53:34,511 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3fe9210195d2e01c: Processing first storage report for DS-d30950a3-de7c-47ec-8c97-d8d05a65bcbf from datanode adb8a7fe-aea7-4258-9a4f-d23be48afc65 2023-04-05 18:53:34,513 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3fe9210195d2e01c: from storage DS-d30950a3-de7c-47ec-8c97-d8d05a65bcbf node DatanodeRegistration(127.0.0.1:42273, datanodeUuid=adb8a7fe-aea7-4258-9a4f-d23be48afc65, infoPort=45811, infoSecurePort=0, ipcPort=43705, storageInfo=lv=-57;cid=testClusterID;nsid=188488759;c=1680720812184), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-05 18:53:34,513 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3fe9210195d2e01c: Processing first storage report for DS-d5a280a0-6ba1-478c-b41f-fc2f0d6a2b23 from datanode adb8a7fe-aea7-4258-9a4f-d23be48afc65 2023-04-05 18:53:34,513 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3fe9210195d2e01c: from storage DS-d5a280a0-6ba1-478c-b41f-fc2f0d6a2b23 node DatanodeRegistration(127.0.0.1:42273, datanodeUuid=adb8a7fe-aea7-4258-9a4f-d23be48afc65, infoPort=45811, infoSecurePort=0, ipcPort=43705, storageInfo=lv=-57;cid=testClusterID;nsid=188488759;c=1680720812184), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-05 18:53:34,573 INFO [Listener at localhost/43705] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/cluster_e107b28c-27c6-6ca4-18a0-71d30edffa22/zookeeper_0, clientPort=49755, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/cluster_e107b28c-27c6-6ca4-18a0-71d30edffa22/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/cluster_e107b28c-27c6-6ca4-18a0-71d30edffa22/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-05 18:53:34,586 INFO [Listener at localhost/43705] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=49755 2023-04-05 18:53:34,593 INFO [Listener at localhost/43705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 18:53:34,595 INFO [Listener at localhost/43705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 18:53:35,262 INFO [Listener at localhost/43705] util.FSUtils(471): Created version file at hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4 with version=8 2023-04-05 18:53:35,262 INFO [Listener at localhost/43705] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/hbase-staging 2023-04-05 18:53:35,564 INFO [Listener at localhost/43705] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-05 18:53:36,025 INFO [Listener at localhost/43705] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-05 18:53:36,065 INFO [Listener at localhost/43705] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 18:53:36,066 INFO [Listener at localhost/43705] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-05 18:53:36,066 INFO [Listener at localhost/43705] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-05 18:53:36,067 INFO [Listener at localhost/43705] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 18:53:36,067 INFO [Listener at localhost/43705] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-05 18:53:36,216 INFO [Listener at localhost/43705] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-05 18:53:36,288 DEBUG [Listener at localhost/43705] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-05 18:53:36,405 INFO [Listener at localhost/43705] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39855 2023-04-05 18:53:36,415 INFO [Listener at localhost/43705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 18:53:36,418 INFO [Listener at localhost/43705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 18:53:36,438 INFO [Listener at localhost/43705] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39855 connecting to ZooKeeper ensemble=127.0.0.1:49755 2023-04-05 18:53:36,476 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:398550x0, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-05 18:53:36,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39855-0x10155ac4ba30000 connected 2023-04-05 18:53:36,504 DEBUG [Listener at localhost/43705] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-05 18:53:36,505 DEBUG [Listener at localhost/43705] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 18:53:36,508 DEBUG [Listener at localhost/43705] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-05 18:53:36,519 DEBUG [Listener at localhost/43705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39855 2023-04-05 18:53:36,520 DEBUG [Listener at localhost/43705] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39855 2023-04-05 18:53:36,520 DEBUG [Listener at localhost/43705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39855 2023-04-05 18:53:36,521 DEBUG [Listener at localhost/43705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39855 2023-04-05 18:53:36,521 DEBUG [Listener at localhost/43705] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39855 2023-04-05 18:53:36,527 INFO [Listener at localhost/43705] master.HMaster(444): hbase.rootdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4, hbase.cluster.distributed=false 2023-04-05 18:53:36,596 INFO [Listener at localhost/43705] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-05 18:53:36,596 INFO [Listener at localhost/43705] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 18:53:36,596 INFO [Listener at localhost/43705] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-05 18:53:36,596 INFO [Listener at localhost/43705] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-05 18:53:36,596 INFO [Listener at localhost/43705] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 18:53:36,597 INFO [Listener at localhost/43705] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-05 18:53:36,601 INFO [Listener at localhost/43705] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-05 18:53:36,604 INFO [Listener at localhost/43705] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33489 2023-04-05 18:53:36,606 INFO [Listener at localhost/43705] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-05 18:53:36,612 DEBUG [Listener at localhost/43705] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-05 18:53:36,614 INFO [Listener at localhost/43705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 18:53:36,617 INFO [Listener at localhost/43705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 18:53:36,619 INFO [Listener at localhost/43705] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33489 connecting to ZooKeeper ensemble=127.0.0.1:49755 2023-04-05 18:53:36,622 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:334890x0, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-05 18:53:36,623 DEBUG [Listener at localhost/43705] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-05 18:53:36,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33489-0x10155ac4ba30001 connected 2023-04-05 18:53:36,624 DEBUG [Listener at localhost/43705] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 18:53:36,625 DEBUG [Listener at localhost/43705] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-05 18:53:36,626 DEBUG [Listener at localhost/43705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33489 2023-04-05 18:53:36,626 DEBUG [Listener at localhost/43705] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33489 2023-04-05 18:53:36,627 DEBUG [Listener at localhost/43705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33489 2023-04-05 18:53:36,627 DEBUG [Listener at localhost/43705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33489 2023-04-05 18:53:36,627 DEBUG [Listener at localhost/43705] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33489 2023-04-05 18:53:36,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39855,1680720815401 2023-04-05 18:53:36,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-05 18:53:36,645 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39855,1680720815401 2023-04-05 18:53:36,663 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-05 18:53:36,663 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-05 18:53:36,663 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 18:53:36,665 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-05 18:53:36,666 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39855,1680720815401 from backup master directory 2023-04-05 18:53:36,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-05 18:53:36,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39855,1680720815401 2023-04-05 18:53:36,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-05 18:53:36,669 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-05 18:53:36,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,39855,1680720815401 2023-04-05 18:53:36,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-05 18:53:36,674 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-05 18:53:36,767 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/hbase.id with ID: 2338699c-fae8-44ce-878d-fc7e35235c03 2023-04-05 18:53:36,809 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 18:53:36,823 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 18:53:36,863 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x30327639 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:36,897 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8f54551, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:36,923 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 18:53:36,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-05 18:53:36,945 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-05 18:53:36,945 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-05 18:53:36,947 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:53:36,951 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:53:36,952 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-05 18:53:36,987 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/data/master/store-tmp 2023-04-05 18:53:37,014 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:53:37,015 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-05 18:53:37,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 18:53:37,015 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 18:53:37,015 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-05 18:53:37,015 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 18:53:37,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 18:53:37,015 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-05 18:53:37,017 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/WALs/jenkins-hbase4.apache.org,39855,1680720815401 2023-04-05 18:53:37,038 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39855%2C1680720815401, suffix=, logDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/WALs/jenkins-hbase4.apache.org,39855,1680720815401, archiveDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/oldWALs, maxLogs=10 2023-04-05 18:53:37,095 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42273,DS-d30950a3-de7c-47ec-8c97-d8d05a65bcbf,DISK] 2023-04-05 18:53:37,105 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:53:37,163 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/WALs/jenkins-hbase4.apache.org,39855,1680720815401/jenkins-hbase4.apache.org%2C39855%2C1680720815401.1680720817048 2023-04-05 18:53:37,164 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42273,DS-d30950a3-de7c-47ec-8c97-d8d05a65bcbf,DISK]] 2023-04-05 18:53:37,164 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:53:37,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:53:37,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-05 18:53:37,169 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-05 18:53:37,222 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-05 18:53:37,229 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-05 18:53:37,251 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-05 18:53:37,263 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:37,269 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-05 18:53:37,271 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-05 18:53:37,286 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-05 18:53:37,290 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 18:53:37,291 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63072576, jitterRate=-0.06014537811279297}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-05 18:53:37,292 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-05 18:53:37,293 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-05 18:53:37,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-05 18:53:37,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-05 18:53:37,314 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-05 18:53:37,315 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-05 18:53:37,346 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-04-05 18:53:37,347 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-05 18:53:37,371 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-05 18:53:37,376 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-05 18:53:37,399 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-05 18:53:37,403 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-05 18:53:37,405 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-05 18:53:37,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-05 18:53:37,414 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-05 18:53:37,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 18:53:37,418 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-05 18:53:37,418 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-05 18:53:37,429 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-05 18:53:37,435 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-05 18:53:37,435 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-05 18:53:37,435 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 18:53:37,436 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,39855,1680720815401, sessionid=0x10155ac4ba30000, setting cluster-up flag (Was=false) 2023-04-05 18:53:37,450 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 18:53:37,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-05 18:53:37,457 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39855,1680720815401 2023-04-05 18:53:37,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 18:53:37,467 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-05 18:53:37,468 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39855,1680720815401 2023-04-05 18:53:37,470 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.hbase-snapshot/.tmp 2023-04-05 18:53:37,537 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(952): ClusterId : 2338699c-fae8-44ce-878d-fc7e35235c03 2023-04-05 18:53:37,542 DEBUG [RS:0;jenkins-hbase4:33489] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-05 18:53:37,548 DEBUG [RS:0;jenkins-hbase4:33489] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-05 18:53:37,548 DEBUG [RS:0;jenkins-hbase4:33489] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-05 18:53:37,553 DEBUG [RS:0;jenkins-hbase4:33489] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-05 18:53:37,554 DEBUG [RS:0;jenkins-hbase4:33489] zookeeper.ReadOnlyZKClient(139): Connect 0x6ecd2ede to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:37,562 DEBUG [RS:0;jenkins-hbase4:33489] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@600acd77, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:37,563 DEBUG [RS:0;jenkins-hbase4:33489] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@643545ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-05 18:53:37,587 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:33489 2023-04-05 18:53:37,591 INFO [RS:0;jenkins-hbase4:33489] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-05 18:53:37,591 INFO [RS:0;jenkins-hbase4:33489] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-05 18:53:37,591 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1023): About to register with Master. 2023-04-05 18:53:37,594 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-05 18:53:37,594 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39855,1680720815401 with isa=jenkins-hbase4.apache.org/172.31.14.131:33489, startcode=1680720816595 2023-04-05 18:53:37,608 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-05 18:53:37,608 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-05 18:53:37,608 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-05 18:53:37,608 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-05 18:53:37,609 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-05 18:53:37,609 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,609 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-05 18:53:37,609 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,611 DEBUG [RS:0;jenkins-hbase4:33489] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-05 18:53:37,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680720847614 2023-04-05 18:53:37,617 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-05 18:53:37,622 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-05 18:53:37,622 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-05 18:53:37,630 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-05 18:53:37,636 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-05 18:53:37,640 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-05 18:53:37,640 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-05 18:53:37,641 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-05 18:53:37,641 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-05 18:53:37,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:37,649 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-05 18:53:37,651 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-05 18:53:37,651 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-05 18:53:37,657 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-05 18:53:37,658 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-05 18:53:37,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680720817659,5,FailOnTimeoutGroup] 2023-04-05 18:53:37,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680720817660,5,FailOnTimeoutGroup] 2023-04-05 18:53:37,660 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:37,660 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-05 18:53:37,661 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:37,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:37,679 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59115, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-05 18:53:37,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:37,704 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4 2023-04-05 18:53:37,704 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:38439 2023-04-05 18:53:37,705 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-05 18:53:37,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-05 18:53:37,711 DEBUG [RS:0;jenkins-hbase4:33489] zookeeper.ZKUtil(162): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:37,711 WARN [RS:0;jenkins-hbase4:33489] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-05 18:53:37,712 INFO [RS:0;jenkins-hbase4:33489] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-05 18:53:37,713 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1947): logDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/WALs/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:37,717 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33489,1680720816595] 2023-04-05 18:53:37,724 DEBUG [RS:0;jenkins-hbase4:33489] zookeeper.ZKUtil(162): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:37,737 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-05 18:53:37,749 INFO [RS:0;jenkins-hbase4:33489] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-05 18:53:37,773 INFO [RS:0;jenkins-hbase4:33489] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-05 18:53:37,776 INFO [RS:0;jenkins-hbase4:33489] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-05 18:53:37,776 INFO [RS:0;jenkins-hbase4:33489] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:37,778 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-05 18:53:37,786 INFO [RS:0;jenkins-hbase4:33489] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:37,786 DEBUG [RS:0;jenkins-hbase4:33489] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,786 DEBUG [RS:0;jenkins-hbase4:33489] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,786 DEBUG [RS:0;jenkins-hbase4:33489] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,787 DEBUG [RS:0;jenkins-hbase4:33489] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,787 DEBUG [RS:0;jenkins-hbase4:33489] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,787 DEBUG [RS:0;jenkins-hbase4:33489] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-05 18:53:37,787 DEBUG [RS:0;jenkins-hbase4:33489] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,787 DEBUG [RS:0;jenkins-hbase4:33489] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,787 DEBUG [RS:0;jenkins-hbase4:33489] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,787 DEBUG [RS:0;jenkins-hbase4:33489] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-05 18:53:37,788 INFO [RS:0;jenkins-hbase4:33489] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:37,788 INFO [RS:0;jenkins-hbase4:33489] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:37,788 INFO [RS:0;jenkins-hbase4:33489] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:37,808 INFO [RS:0;jenkins-hbase4:33489] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-05 18:53:37,811 INFO [RS:0;jenkins-hbase4:33489] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33489,1680720816595-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:37,829 INFO [RS:0;jenkins-hbase4:33489] regionserver.Replication(203): jenkins-hbase4.apache.org,33489,1680720816595 started 2023-04-05 18:53:37,829 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33489,1680720816595, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33489, sessionid=0x10155ac4ba30001 2023-04-05 18:53:37,829 DEBUG [RS:0;jenkins-hbase4:33489] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-05 18:53:37,829 DEBUG [RS:0;jenkins-hbase4:33489] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:37,829 DEBUG [RS:0;jenkins-hbase4:33489] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33489,1680720816595' 2023-04-05 18:53:37,830 DEBUG [RS:0;jenkins-hbase4:33489] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:37,830 DEBUG [RS:0;jenkins-hbase4:33489] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:37,831 DEBUG [RS:0;jenkins-hbase4:33489] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-05 18:53:37,831 DEBUG [RS:0;jenkins-hbase4:33489] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-05 18:53:37,831 DEBUG [RS:0;jenkins-hbase4:33489] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:37,831 DEBUG [RS:0;jenkins-hbase4:33489] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33489,1680720816595' 2023-04-05 18:53:37,831 DEBUG [RS:0;jenkins-hbase4:33489] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-05 18:53:37,832 DEBUG [RS:0;jenkins-hbase4:33489] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-05 18:53:37,832 DEBUG [RS:0;jenkins-hbase4:33489] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-05 18:53:37,833 INFO [RS:0;jenkins-hbase4:33489] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-05 18:53:37,833 INFO [RS:0;jenkins-hbase4:33489] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-05 18:53:37,944 INFO [RS:0;jenkins-hbase4:33489] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33489%2C1680720816595, suffix=, logDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/WALs/jenkins-hbase4.apache.org,33489,1680720816595, archiveDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/oldWALs, maxLogs=32 2023-04-05 18:53:37,963 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42273,DS-d30950a3-de7c-47ec-8c97-d8d05a65bcbf,DISK] 2023-04-05 18:53:37,966 INFO [RS:0;jenkins-hbase4:33489] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/WALs/jenkins-hbase4.apache.org,33489,1680720816595/jenkins-hbase4.apache.org%2C33489%2C1680720816595.1680720817947 2023-04-05 18:53:37,967 DEBUG [RS:0;jenkins-hbase4:33489] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42273,DS-d30950a3-de7c-47ec-8c97-d8d05a65bcbf,DISK]] 2023-04-05 18:53:38,078 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-05 18:53:38,079 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-05 18:53:38,079 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4 2023-04-05 18:53:38,096 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:53:38,098 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-05 18:53:38,101 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/info 2023-04-05 18:53:38,102 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-05 18:53:38,104 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:38,104 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-05 18:53:38,107 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/rep_barrier 2023-04-05 18:53:38,108 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-05 18:53:38,109 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:38,109 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-05 18:53:38,112 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/table 2023-04-05 18:53:38,112 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-05 18:53:38,113 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:38,115 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740 2023-04-05 18:53:38,116 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740 2023-04-05 18:53:38,120 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:53:38,122 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-05 18:53:38,125 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 18:53:38,126 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66353600, jitterRate=-0.011254310607910156}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:53:38,127 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-05 18:53:38,127 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-05 18:53:38,127 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-05 18:53:38,127 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-05 18:53:38,127 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-05 18:53:38,127 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-05 18:53:38,129 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-05 18:53:38,129 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-05 18:53:38,134 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-05 18:53:38,134 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-05 18:53:38,142 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-05 18:53:38,156 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-05 18:53:38,159 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-05 18:53:38,311 DEBUG [jenkins-hbase4:39855] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-05 18:53:38,314 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33489,1680720816595, state=OPENING 2023-04-05 18:53:38,321 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-05 18:53:38,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 18:53:38,323 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-05 18:53:38,325 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:53:38,508 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:38,511 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-05 18:53:38,514 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42876, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-05 18:53:38,525 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-05 18:53:38,525 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-05 18:53:38,528 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33489%2C1680720816595.meta, suffix=.meta, logDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/WALs/jenkins-hbase4.apache.org,33489,1680720816595, archiveDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/oldWALs, maxLogs=32 2023-04-05 18:53:38,544 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42273,DS-d30950a3-de7c-47ec-8c97-d8d05a65bcbf,DISK] 2023-04-05 18:53:38,548 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/WALs/jenkins-hbase4.apache.org,33489,1680720816595/jenkins-hbase4.apache.org%2C33489%2C1680720816595.meta.1680720818529.meta 2023-04-05 18:53:38,549 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42273,DS-d30950a3-de7c-47ec-8c97-d8d05a65bcbf,DISK]] 2023-04-05 18:53:38,549 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:53:38,551 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-05 18:53:38,573 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-05 18:53:38,579 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-05 18:53:38,584 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-05 18:53:38,585 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:53:38,585 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-05 18:53:38,585 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-05 18:53:38,588 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-05 18:53:38,589 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/info 2023-04-05 18:53:38,590 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/info 2023-04-05 18:53:38,590 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-05 18:53:38,591 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:38,591 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-05 18:53:38,593 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/rep_barrier 2023-04-05 18:53:38,593 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/rep_barrier 2023-04-05 18:53:38,593 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-05 18:53:38,594 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:38,595 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-05 18:53:38,596 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/table 2023-04-05 18:53:38,596 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/table 2023-04-05 18:53:38,597 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-05 18:53:38,597 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:38,599 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740 2023-04-05 18:53:38,601 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740 2023-04-05 18:53:38,605 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:53:38,608 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-05 18:53:38,610 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73632126, jitterRate=0.09720417857170105}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:53:38,610 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-05 18:53:38,621 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680720818500 2023-04-05 18:53:38,637 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-05 18:53:38,638 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-05 18:53:38,638 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33489,1680720816595, state=OPEN 2023-04-05 18:53:38,641 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-05 18:53:38,641 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-05 18:53:38,645 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-05 18:53:38,645 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33489,1680720816595 in 316 msec 2023-04-05 18:53:38,650 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-05 18:53:38,650 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 504 msec 2023-04-05 18:53:38,657 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1400 sec 2023-04-05 18:53:38,657 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680720818657, completionTime=-1 2023-04-05 18:53:38,657 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-05 18:53:38,658 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-05 18:53:38,718 DEBUG [hconnection-0x5fffeea1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:38,721 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42878, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:38,738 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-05 18:53:38,738 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680720878738 2023-04-05 18:53:38,738 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680720938738 2023-04-05 18:53:38,738 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 80 msec 2023-04-05 18:53:38,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39855,1680720815401-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:38,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39855,1680720815401-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:38,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39855,1680720815401-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:38,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39855, period=300000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:38,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-05 18:53:38,768 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-05 18:53:38,776 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-05 18:53:38,777 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-05 18:53:38,787 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-05 18:53:38,789 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 18:53:38,792 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 18:53:38,812 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:53:38,814 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78 empty. 2023-04-05 18:53:38,815 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:53:38,815 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-05 18:53:39,242 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-05 18:53:39,245 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 482b8a813b86c6c27617f295eb5f6e78, NAME => 'hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp 2023-04-05 18:53:39,260 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:53:39,260 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 482b8a813b86c6c27617f295eb5f6e78, disabling compactions & flushes 2023-04-05 18:53:39,260 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:53:39,260 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:53:39,260 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. after waiting 0 ms 2023-04-05 18:53:39,260 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:53:39,260 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:53:39,260 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 482b8a813b86c6c27617f295eb5f6e78: 2023-04-05 18:53:39,265 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 18:53:39,280 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680720819268"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720819268"}]},"ts":"1680720819268"} 2023-04-05 18:53:39,306 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 18:53:39,308 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 18:53:39,313 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720819308"}]},"ts":"1680720819308"} 2023-04-05 18:53:39,317 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-05 18:53:39,326 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=482b8a813b86c6c27617f295eb5f6e78, ASSIGN}] 2023-04-05 18:53:39,329 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=482b8a813b86c6c27617f295eb5f6e78, ASSIGN 2023-04-05 18:53:39,330 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=482b8a813b86c6c27617f295eb5f6e78, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33489,1680720816595; forceNewPlan=false, retain=false 2023-04-05 18:53:39,481 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=482b8a813b86c6c27617f295eb5f6e78, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:39,482 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680720819481"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720819481"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720819481"}]},"ts":"1680720819481"} 2023-04-05 18:53:39,489 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 482b8a813b86c6c27617f295eb5f6e78, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:53:39,649 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:53:39,650 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 482b8a813b86c6c27617f295eb5f6e78, NAME => 'hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:53:39,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:53:39,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:53:39,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:53:39,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:53:39,653 INFO [StoreOpener-482b8a813b86c6c27617f295eb5f6e78-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:53:39,656 DEBUG [StoreOpener-482b8a813b86c6c27617f295eb5f6e78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78/info 2023-04-05 18:53:39,656 DEBUG [StoreOpener-482b8a813b86c6c27617f295eb5f6e78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78/info 2023-04-05 18:53:39,657 INFO [StoreOpener-482b8a813b86c6c27617f295eb5f6e78-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 482b8a813b86c6c27617f295eb5f6e78 columnFamilyName info 2023-04-05 18:53:39,658 INFO [StoreOpener-482b8a813b86c6c27617f295eb5f6e78-1] regionserver.HStore(310): Store=482b8a813b86c6c27617f295eb5f6e78/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:39,660 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:53:39,662 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:53:39,667 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:53:39,672 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 18:53:39,673 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 482b8a813b86c6c27617f295eb5f6e78; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68052632, jitterRate=0.014063239097595215}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-05 18:53:39,673 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 482b8a813b86c6c27617f295eb5f6e78: 2023-04-05 18:53:39,675 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78., pid=6, masterSystemTime=1680720819643 2023-04-05 18:53:39,679 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:53:39,679 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:53:39,681 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=482b8a813b86c6c27617f295eb5f6e78, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:39,681 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680720819680"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680720819680"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680720819680"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680720819680"}]},"ts":"1680720819680"} 2023-04-05 18:53:39,689 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-05 18:53:39,689 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 482b8a813b86c6c27617f295eb5f6e78, server=jenkins-hbase4.apache.org,33489,1680720816595 in 196 msec 2023-04-05 18:53:39,693 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-05 18:53:39,694 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=482b8a813b86c6c27617f295eb5f6e78, ASSIGN in 363 msec 2023-04-05 18:53:39,695 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 18:53:39,696 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720819695"}]},"ts":"1680720819695"} 2023-04-05 18:53:39,698 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-05 18:53:39,702 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 18:53:39,704 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 924 msec 2023-04-05 18:53:39,790 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-05 18:53:39,791 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-05 18:53:39,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 18:53:39,830 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-05 18:53:39,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-05 18:53:39,852 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 33 msec 2023-04-05 18:53:39,863 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-05 18:53:39,877 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-05 18:53:39,882 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-04-05 18:53:39,889 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-05 18:53:39,893 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-05 18:53:39,893 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.223sec 2023-04-05 18:53:39,894 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-05 18:53:39,896 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-05 18:53:39,896 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-05 18:53:39,897 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39855,1680720815401-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-05 18:53:39,898 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39855,1680720815401-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-05 18:53:39,908 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-05 18:53:39,945 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x643ba65c to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:39,950 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28099d00, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:39,962 DEBUG [hconnection-0x20d652f1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:39,973 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42882, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:39,983 INFO [Listener at localhost/43705] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39855,1680720815401 2023-04-05 18:53:40,004 INFO [Listener at localhost/43705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=297, ProcessCount=170, AvailableMemoryMB=2317 2023-04-05 18:53:40,019 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 18:53:40,023 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52478, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 18:53:40,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 18:53:40,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 18:53:40,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 18:53:40,039 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 18:53:40,041 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 18:53:40,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-05 18:53:40,044 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,045 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5 empty. 2023-04-05 18:53:40,047 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,047 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:53:40,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 18:53:40,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 18:53:40,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 18:53:40,493 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 18:53:40,495 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => be0aa03376eb89131764dd34306ebfb5, NAME => 'TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp 2023-04-05 18:53:40,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:53:40,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing be0aa03376eb89131764dd34306ebfb5, disabling compactions & flushes 2023-04-05 18:53:40,506 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:40,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:40,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. after waiting 0 ms 2023-04-05 18:53:40,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:40,506 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:40,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:40,510 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 18:53:40,512 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720820512"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720820512"}]},"ts":"1680720820512"} 2023-04-05 18:53:40,515 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 18:53:40,516 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 18:53:40,516 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720820516"}]},"ts":"1680720820516"} 2023-04-05 18:53:40,518 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 18:53:40,522 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=be0aa03376eb89131764dd34306ebfb5, ASSIGN}] 2023-04-05 18:53:40,524 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=be0aa03376eb89131764dd34306ebfb5, ASSIGN 2023-04-05 18:53:40,525 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=be0aa03376eb89131764dd34306ebfb5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33489,1680720816595; forceNewPlan=false, retain=false 2023-04-05 18:53:40,676 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=be0aa03376eb89131764dd34306ebfb5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:40,677 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720820676"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720820676"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720820676"}]},"ts":"1680720820676"} 2023-04-05 18:53:40,680 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:53:40,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 18:53:40,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:40,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => be0aa03376eb89131764dd34306ebfb5, NAME => 'TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:53:40,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:53:40,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,843 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,845 DEBUG [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A 2023-04-05 18:53:40,845 DEBUG [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A 2023-04-05 18:53:40,846 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:53:40,847 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be0aa03376eb89131764dd34306ebfb5 columnFamilyName A 2023-04-05 18:53:40,848 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] regionserver.HStore(310): Store=be0aa03376eb89131764dd34306ebfb5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:40,848 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,850 DEBUG [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B 2023-04-05 18:53:40,850 DEBUG [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B 2023-04-05 18:53:40,850 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:53:40,851 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be0aa03376eb89131764dd34306ebfb5 columnFamilyName B 2023-04-05 18:53:40,851 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] regionserver.HStore(310): Store=be0aa03376eb89131764dd34306ebfb5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:40,851 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,854 DEBUG [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C 2023-04-05 18:53:40,854 DEBUG [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C 2023-04-05 18:53:40,854 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:53:40,854 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be0aa03376eb89131764dd34306ebfb5 columnFamilyName C 2023-04-05 18:53:40,855 INFO [StoreOpener-be0aa03376eb89131764dd34306ebfb5-1] regionserver.HStore(310): Store=be0aa03376eb89131764dd34306ebfb5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:53:40,856 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:40,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:53:40,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:40,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 18:53:40,870 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened be0aa03376eb89131764dd34306ebfb5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72251674, jitterRate=0.07663384079933167}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:53:40,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:40,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., pid=11, masterSystemTime=1680720820834 2023-04-05 18:53:40,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:40,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:40,876 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=be0aa03376eb89131764dd34306ebfb5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:40,876 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720820876"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680720820876"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680720820876"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680720820876"}]},"ts":"1680720820876"} 2023-04-05 18:53:40,882 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-05 18:53:40,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 in 199 msec 2023-04-05 18:53:40,886 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-05 18:53:40,887 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=be0aa03376eb89131764dd34306ebfb5, ASSIGN in 360 msec 2023-04-05 18:53:40,888 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 18:53:40,888 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720820888"}]},"ts":"1680720820888"} 2023-04-05 18:53:40,890 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 18:53:40,894 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 18:53:40,896 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 859 msec 2023-04-05 18:53:41,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 18:53:41,194 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-05 18:53:41,198 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x4281aac2 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:41,203 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@762c42ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:41,206 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:41,209 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42890, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:41,212 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 18:53:41,214 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52490, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 18:53:41,222 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x5e91d243 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:41,226 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:41,227 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:41,230 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e728705, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:41,231 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x46192d17 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:41,236 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:41,236 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:41,240 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42927198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:41,240 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x49a45b6d to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:41,244 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1818ad0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:41,245 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x2879e2e1 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:41,248 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cd3abbd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:41,249 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:41,280 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:41,281 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x4bd6fc59 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:41,285 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4808c6d5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:41,285 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x4fab6dfa to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:53:41,289 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33637462, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:53:41,295 DEBUG [hconnection-0x2b3ecf8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:41,299 DEBUG [hconnection-0x524c453b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:41,299 DEBUG [hconnection-0x4d7da3d0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:41,305 DEBUG [hconnection-0x6881f65f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:41,308 DEBUG [hconnection-0x38d4eae1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:41,309 DEBUG [hconnection-0x1653b219-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:41,309 DEBUG [hconnection-0x6d041e91-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:41,315 DEBUG [hconnection-0x198ec8c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:41,318 DEBUG [hconnection-0x4c2fffd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:53:41,319 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42912, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:41,323 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42922, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:41,323 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42894, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:41,323 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42904, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:41,323 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42918, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:41,324 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42946, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:41,324 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42930, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:41,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:41,342 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42958, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:41,343 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42956, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:53:41,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:41,360 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:41,360 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:41,361 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:41,361 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:41,361 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:41,362 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:41,363 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:41,363 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:41,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:41,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:41,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:41,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:41,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:41,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:41,376 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:41,378 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:41,378 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:41,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:41,380 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:41,381 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-05 18:53:41,398 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:41,399 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:41,399 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:41,399 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:41,399 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:41,399 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:41,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:41,412 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:41,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720821468, max timestamp=1680720821493Num uniques 859; 2023-04-05 18:53:41,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:53:41,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720821468, max timestamp=1680720821512Num uniques 859; 2023-04-05 18:53:41,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:53:41,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720821468, max timestamp=1680720821493Num uniques 859; 2023-04-05 18:53:41,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:53:41,548 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/e3574efb46e94b9e9aad23a41b6276a1 2023-04-05 18:53:41,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720821588, max timestamp=1680720821599Num uniques 1018; 2023-04-05 18:53:41,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 18:53:41,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720821588, max timestamp=1680720821599Num uniques 1018; 2023-04-05 18:53:41,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-05 18:53:41,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720821588, max timestamp=1680720821596Num uniques 1018; 2023-04-05 18:53:41,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-05 18:53:41,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:41,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720881662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720821647, max timestamp=1680720821663Num uniques 1009; 2023-04-05 18:53:41,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-05 18:53:41,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720821647, max timestamp=1680720821665Num uniques 1027; 2023-04-05 18:53:41,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-05 18:53:41,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720821647, max timestamp=1680720821666Num uniques 1009; 2023-04-05 18:53:41,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-05 18:53:41,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680720821703, max timestamp=1680720821729Num uniques 1009; 2023-04-05 18:53:41,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13096, suffix heap size=132904,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=286 2023-04-05 18:53:41,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720821703, max timestamp=1680720821729Num uniques 1009; 2023-04-05 18:53:41,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-05 18:53:41,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720821703, max timestamp=1680720821729Num uniques 1009; 2023-04-05 18:53:41,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-05 18:53:41,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680720821753, max timestamp=1680720821781Num uniques 1009; 2023-04-05 18:53:41,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680720821753, max timestamp=1680720821781Num uniques 1145; 2023-04-05 18:53:41,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=18090, suffix heap size=147048,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=395 2023-04-05 18:53:41,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=18090, suffix heap size=132904,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=395 2023-04-05 18:53:41,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720821772, max timestamp=1680720821781Num uniques 1009; 2023-04-05 18:53:41,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-05 18:53:41,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:41,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720881817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:41,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:41,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720881817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720881820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:41,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720881820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:41,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720881833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:41,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720881926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:41,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720881927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:41,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720881927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:41,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:41,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720881928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720882037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,054 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/88a430b42cf649d8a4239283a2824245 2023-04-05 18:53:42,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720882141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720882143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720882143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720882143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,150 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/3b741579c73d4381b3aa71013d10d592 2023-04-05 18:53:42,167 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/e3574efb46e94b9e9aad23a41b6276a1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/e3574efb46e94b9e9aad23a41b6276a1 2023-04-05 18:53:42,180 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/e3574efb46e94b9e9aad23a41b6276a1, entries=150, sequenceid=9, filesize=11.6 K 2023-04-05 18:53:42,189 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/88a430b42cf649d8a4239283a2824245 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/88a430b42cf649d8a4239283a2824245 2023-04-05 18:53:42,213 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/88a430b42cf649d8a4239283a2824245, entries=150, sequenceid=9, filesize=11.6 K 2023-04-05 18:53:42,216 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/3b741579c73d4381b3aa71013d10d592 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3b741579c73d4381b3aa71013d10d592 2023-04-05 18:53:42,251 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3b741579c73d4381b3aa71013d10d592, entries=150, sequenceid=9, filesize=11.6 K 2023-04-05 18:53:42,259 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=156.77 KB/160530 for be0aa03376eb89131764dd34306ebfb5 in 878ms, sequenceid=9, compaction requested=false 2023-04-05 18:53:42,275 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-05 18:53:42,278 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:42,278 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:42,279 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:42,281 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:42,281 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,281 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:42,281 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:42,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,284 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:42,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a5fe95c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:42,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:42,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,292 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:42,294 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,294 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:42,298 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:42,298 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,298 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,298 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:42,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:42,305 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:42,306 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:42,306 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:42,306 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:42,306 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:42,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:42,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:42,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,316 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,317 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:42,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:42,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,352 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,352 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:42,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:42,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:42,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:42,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:42,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:42,375 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:42,375 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:42,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:42,390 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=163.48 KB heapSize=458.21 KB 2023-04-05 18:53:42,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:42,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:53:42,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:42,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:53:42,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:42,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:53:42,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:42,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720882449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720882452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720882453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720882454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720882456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,475 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:42,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:42,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/15b961f2c13e449898ee0d0464449d11 2023-04-05 18:53:42,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:42,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:42,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:42,528 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:42,528 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:42,531 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:42,532 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:42,532 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,532 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,535 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:42,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:42,536 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,536 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:42,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:42,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:42,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:42,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:42,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:42,540 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:42,551 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:42,551 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:42,551 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:42,551 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:42,556 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:42,556 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:42,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:42,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:42,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:42,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:42,561 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,562 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,565 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:42,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54283385[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:42,567 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:42,567 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:42,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:42,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:42,577 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:42,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:42,587 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:42,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/c75d942efdf243caa616063d18ff79f5 2023-04-05 18:53:42,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720882587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:42,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:42,599 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:42,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:42,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:42,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:42,599 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:42,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:42,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:42,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:42,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,610 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,614 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:42,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:42,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:42,628 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:42,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:42,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,629 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,629 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,629 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:42,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:42,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:42,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:42,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:42,630 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:42,630 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:42,730 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:42,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:42,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:42,774 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:42,774 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:42,775 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:42,775 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:42,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,778 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:42,778 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:42,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:42,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:42,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:42,786 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:42,786 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:42,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:42,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:42,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:42,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:42,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:42,791 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:42,791 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:42,791 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,791 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:42,791 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:42,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,796 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:42,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477fc28f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:42,801 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:42,801 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720882800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,802 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:42,803 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,803 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:42,805 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,805 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:42,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:42,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:42,813 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:42,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:42,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:42,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:42,814 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:42,816 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,816 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:42,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,816 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:42,817 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:42,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:42,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:42,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:42,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:42,836 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:42,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:42,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,837 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:42,842 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,842 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:42,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:42,842 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,942 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:42,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:42,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:42,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:42,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:42,955 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:42,955 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:42,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:42,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:42,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720882959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,962 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,962 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:42,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:42,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:42,962 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,963 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:42,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:42,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:42,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:42,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:42,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:42,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720882961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:42,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:42,966 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:42,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720882962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:42,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:42,972 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:42,972 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:42,972 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:42,972 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:42,974 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:42,974 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:42,974 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,974 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:42,974 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:42,977 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:42,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77c09868[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:42,981 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:42,981 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,983 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:42,983 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:42,983 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:42,983 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:42,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:42,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:42,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720882978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:42,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:42,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:42,992 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:42,992 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:42,992 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:42,992 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:42,992 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:42,995 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:42,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:43,000 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:43,000 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:43,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:43,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:43,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:43,003 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:43,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:43,015 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:43,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:43,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:43,016 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:43,016 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:43,016 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:43,016 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:43,020 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:43,020 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:43,020 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:43,020 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:43,020 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:43,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:43,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:43,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:43,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:43,024 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:43,024 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:43,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=47.19 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/bdcb800ee1754159bf3f1a46f931af3b 2023-04-05 18:53:43,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/15b961f2c13e449898ee0d0464449d11 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/15b961f2c13e449898ee0d0464449d11 2023-04-05 18:53:43,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/15b961f2c13e449898ee0d0464449d11, entries=200, sequenceid=116, filesize=13.9 K 2023-04-05 18:53:43,108 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/c75d942efdf243caa616063d18ff79f5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/c75d942efdf243caa616063d18ff79f5 2023-04-05 18:53:43,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:43,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720883106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:43,124 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:43,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/c75d942efdf243caa616063d18ff79f5, entries=150, sequenceid=116, filesize=11.6 K 2023-04-05 18:53:43,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:43,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/bdcb800ee1754159bf3f1a46f931af3b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bdcb800ee1754159bf3f1a46f931af3b 2023-04-05 18:53:43,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:43,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:43,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:43,152 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:43,152 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:43,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:43,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:43,155 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:43,155 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:43,158 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:43,158 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:43,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:43,159 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:43,159 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:43,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:43,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:43,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:43,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:43,161 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:43,161 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:43,163 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:43,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bdcb800ee1754159bf3f1a46f931af3b, entries=150, sequenceid=116, filesize=11.6 K 2023-04-05 18:53:43,164 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:43,164 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:43,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~163.48 KB/167400, heapSize ~458.16 KB/469160, currentSize=20.13 KB/20610 for be0aa03376eb89131764dd34306ebfb5 in 774ms, sequenceid=116, compaction requested=false 2023-04-05 18:53:43,165 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:43,167 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:43,167 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:43,168 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-05 18:53:43,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:43,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:43,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:43,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:43,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:43,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:43,226 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/ae16fae4ab1343248da2beb73866b6fa 2023-04-05 18:53:43,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:43,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:43,692 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/dc9c73a4a31d411794f6c5eb8654728a 2023-04-05 18:53:43,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720823739, max timestamp=1680720823765Num uniques 859; 2023-04-05 18:53:43,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720823739, max timestamp=1680720823765Num uniques 859; 2023-04-05 18:53:43,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720823739, max timestamp=1680720823765Num uniques 859; 2023-04-05 18:53:43,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:53:43,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:53:43,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:53:43,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720823806, max timestamp=1680720823848Num uniques 1018; 2023-04-05 18:53:43,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 18:53:43,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720823806, max timestamp=1680720823848Num uniques 1018; 2023-04-05 18:53:43,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 18:53:43,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720823806, max timestamp=1680720823848Num uniques 1018; 2023-04-05 18:53:43,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 18:53:43,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680720823918, max timestamp=1680720823936Num uniques 1077; 2023-04-05 18:53:43,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-05 18:53:43,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680720823918, max timestamp=1680720823936Num uniques 1077; 2023-04-05 18:53:43,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-05 18:53:43,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680720823918, max timestamp=1680720823936Num uniques 1077; 2023-04-05 18:53:43,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-05 18:53:43,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680720823980, max timestamp=1680720823992Num uniques 1036; 2023-04-05 18:53:43,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680720823980, max timestamp=1680720823992Num uniques 1036; 2023-04-05 18:53:43,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-05 18:53:43,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680720823982, max timestamp=1680720823992Num uniques 1036; 2023-04-05 18:53:43,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 18:53:43,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-05 18:53:44,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680720824013, max timestamp=1680720824024Num uniques 1195; 2023-04-05 18:53:44,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-05 18:53:44,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680720824013, max timestamp=1680720824024Num uniques 1045; 2023-04-05 18:53:44,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 18:53:44,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680720824013, max timestamp=1680720824024Num uniques 1195; 2023-04-05 18:53:44,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-05 18:53:44,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1680720824055, max timestamp=1680720824064Num uniques 1054; 2023-04-05 18:53:44,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=9334, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-05 18:53:44,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1680720824055, max timestamp=1680720824064Num uniques 1054; 2023-04-05 18:53:44,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=9334, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-05 18:53:44,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680720824055, max timestamp=1680720824064Num uniques 1054; 2023-04-05 18:53:44,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-05 18:53:44,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680720824087, max timestamp=1680720824096Num uniques 1063; 2023-04-05 18:53:44,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680720824087, max timestamp=1680720824096Num uniques 1063; 2023-04-05 18:53:44,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-05 18:53:44,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-05 18:53:44,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680720824087, max timestamp=1680720824096Num uniques 1063; 2023-04-05 18:53:44,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-05 18:53:44,162 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/81b03c1c76de4ecfa2744ba71a9502a6 2023-04-05 18:53:44,173 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 18:53:44,181 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/ae16fae4ab1343248da2beb73866b6fa as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ae16fae4ab1343248da2beb73866b6fa 2023-04-05 18:53:44,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680720824144, max timestamp=1680720824177Num uniques 1122; 2023-04-05 18:53:44,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680720824144, max timestamp=1680720824177Num uniques 1122; 2023-04-05 18:53:44,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-05 18:53:44,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680720824144, max timestamp=1680720824177Num uniques 1122; 2023-04-05 18:53:44,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-05 18:53:44,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-05 18:53:44,203 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ae16fae4ab1343248da2beb73866b6fa, entries=100, sequenceid=122, filesize=9.3 K 2023-04-05 18:53:44,209 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/dc9c73a4a31d411794f6c5eb8654728a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/dc9c73a4a31d411794f6c5eb8654728a 2023-04-05 18:53:44,230 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/dc9c73a4a31d411794f6c5eb8654728a, entries=100, sequenceid=122, filesize=9.3 K 2023-04-05 18:53:44,233 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/81b03c1c76de4ecfa2744ba71a9502a6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/81b03c1c76de4ecfa2744ba71a9502a6 2023-04-05 18:53:44,243 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/81b03c1c76de4ecfa2744ba71a9502a6, entries=100, sequenceid=122, filesize=9.3 K 2023-04-05 18:53:44,255 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=93.93 KB/96180 for be0aa03376eb89131764dd34306ebfb5 in 1088ms, sequenceid=122, compaction requested=true 2023-04-05 18:53:44,256 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:44,256 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:44,256 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:44,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:44,257 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:44,257 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=114.05 KB heapSize=324 KB 2023-04-05 18:53:44,257 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:44,257 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:44,257 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:44,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:44,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:44,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:44,257 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,258 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:44,258 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:44,261 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:44,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62409ac5[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:44,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:44,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:44,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:44,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:44,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:44,283 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:44,283 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:44,284 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:44,284 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:44,284 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:44,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:44,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,294 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,298 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:44,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:44,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:44,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:44,316 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,317 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:44,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:44,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:44,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,318 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:44,318 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:44,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720884293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720884295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720884296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,330 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-05 18:53:44,332 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-05 18:53:44,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720884301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720884318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,418 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:44,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:44,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:44,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:44,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:44,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,424 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:44,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:44,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720884420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,426 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:44,426 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:44,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720884421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,426 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,426 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,429 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:44,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:44,430 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,430 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:44,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:44,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:44,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:44,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:44,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:44,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:44,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:44,434 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:44,434 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:44,434 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:44,434 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:44,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:44,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:44,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:44,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:44,436 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,436 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720884435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,438 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,438 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:44,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720884435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720884438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:44,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77613913[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:44,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:44,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:44,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:44,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:44,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:44,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:44,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:44,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:44,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:44,449 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:44,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:44,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:44,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:44,449 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:44,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:44,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,453 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,457 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:44,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:44,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:44,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:44,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:44,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:44,462 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:44,462 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:44,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:44,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:44,562 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:44,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:44,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:44,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:44,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:44,569 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:44,569 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:44,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:44,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:44,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,572 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:44,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:44,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:44,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:44,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:44,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:44,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:44,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:44,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:44,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:44,576 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:44,576 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:44,576 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:44,576 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:44,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:44,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:44,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:44,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:44,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:44,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb3f6ca[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:44,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:44,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,583 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:44,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:44,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:44,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:44,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:44,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:44,586 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:44,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:44,589 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:44,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:44,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:44,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:44,590 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:44,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:44,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,593 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,596 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,600 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:44,600 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:44,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:44,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:44,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,603 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:44,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:44,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:44,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:44,604 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:44,604 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:44,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720884628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720884628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720884640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720884641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:44,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720884641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,705 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:44,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:44,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:44,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:44,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:44,710 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:44,710 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:44,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:44,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:44,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,718 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,718 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:44,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:44,719 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,719 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:44,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:44,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:44,721 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:44,721 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:44,722 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:44,722 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:44,722 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:44,722 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:44,724 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,724 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,725 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,725 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:44,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4324ac[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:44,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/59cb9a000c444bbe8c390edbb1a72eec 2023-04-05 18:53:44,729 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:44,729 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,730 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,731 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:44,731 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:44,731 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:44,731 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,731 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:44,733 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:44,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:44,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:44,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:44,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:44,738 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:44,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:44,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:44,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:44,738 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:44,741 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,741 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:44,742 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:44,743 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,749 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,767 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:44,767 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:44,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:44,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:44,767 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:44,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:44,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:44,768 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:44,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:44,769 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:44,769 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:44,773 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/0b79b3bab3a54817bbe579e42334bb96 2023-04-05 18:53:44,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/276c7ec8623b48188af47a3173ed7a3f 2023-04-05 18:53:44,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/59cb9a000c444bbe8c390edbb1a72eec as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/59cb9a000c444bbe8c390edbb1a72eec 2023-04-05 18:53:44,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/59cb9a000c444bbe8c390edbb1a72eec, entries=150, sequenceid=276, filesize=11.9 K 2023-04-05 18:53:44,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/0b79b3bab3a54817bbe579e42334bb96 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0b79b3bab3a54817bbe579e42334bb96 2023-04-05 18:53:44,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0b79b3bab3a54817bbe579e42334bb96, entries=150, sequenceid=276, filesize=11.9 K 2023-04-05 18:53:44,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/276c7ec8623b48188af47a3173ed7a3f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/276c7ec8623b48188af47a3173ed7a3f 2023-04-05 18:53:44,862 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/276c7ec8623b48188af47a3173ed7a3f, entries=150, sequenceid=276, filesize=11.9 K 2023-04-05 18:53:44,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~323.95 KB/331728, currentSize=67.09 KB/68700 for be0aa03376eb89131764dd34306ebfb5 in 609ms, sequenceid=276, compaction requested=true 2023-04-05 18:53:44,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:44,868 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:44,868 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:44,868 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:44,869 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:44,869 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:44,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:44,874 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45500 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:44,875 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:44,877 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/B is initiating minor compaction (all files) 2023-04-05 18:53:44,877 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/B in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:44,877 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/88a430b42cf649d8a4239283a2824245, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/c75d942efdf243caa616063d18ff79f5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/dc9c73a4a31d411794f6c5eb8654728a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0b79b3bab3a54817bbe579e42334bb96] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=44.4 K 2023-04-05 18:53:44,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:44,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:44,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:44,881 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:44,884 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:44,890 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:44,890 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:44,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,891 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 88a430b42cf649d8a4239283a2824245, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680720821362 2023-04-05 18:53:44,892 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,892 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:44,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:44,893 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting c75d942efdf243caa616063d18ff79f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680720821799 2023-04-05 18:53:44,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:44,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:44,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:44,894 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting dc9c73a4a31d411794f6c5eb8654728a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680720822425 2023-04-05 18:53:44,895 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47840 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:44,895 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/A is initiating minor compaction (all files) 2023-04-05 18:53:44,895 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0b79b3bab3a54817bbe579e42334bb96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680720824213 2023-04-05 18:53:44,896 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/A in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:44,896 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/e3574efb46e94b9e9aad23a41b6276a1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/15b961f2c13e449898ee0d0464449d11, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ae16fae4ab1343248da2beb73866b6fa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/59cb9a000c444bbe8c390edbb1a72eec] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=46.7 K 2023-04-05 18:53:44,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:44,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:44,896 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:44,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:44,902 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:44,902 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:44,902 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting e3574efb46e94b9e9aad23a41b6276a1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680720821362 2023-04-05 18:53:44,906 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 15b961f2c13e449898ee0d0464449d11, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680720821799 2023-04-05 18:53:44,909 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting ae16fae4ab1343248da2beb73866b6fa, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680720822425 2023-04-05 18:53:44,911 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 59cb9a000c444bbe8c390edbb1a72eec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680720824213 2023-04-05 18:53:44,920 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:44,920 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:44,920 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 18:53:44,920 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:44,921 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:44,921 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:44,921 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:44,921 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:44,921 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:44,943 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#B#compaction#12 average throughput is 0.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:44,943 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:44,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:44,949 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#A#compaction#14 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:44,980 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/3163486607794186b204f79d2e1b51ba 2023-04-05 18:53:45,000 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/490a41f41cf646a48cd087ac64387c56 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/490a41f41cf646a48cd087ac64387c56 2023-04-05 18:53:45,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720885012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720885014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,030 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/4b2c981f2fba455a9a6857a476c7de9c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4b2c981f2fba455a9a6857a476c7de9c 2023-04-05 18:53:45,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720885024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720885026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720885027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,053 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/A of be0aa03376eb89131764dd34306ebfb5 into 490a41f41cf646a48cd087ac64387c56(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:45,053 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:45,053 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/A, priority=12, startTime=1680720824868; duration=0sec 2023-04-05 18:53:45,054 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/B of be0aa03376eb89131764dd34306ebfb5 into 4b2c981f2fba455a9a6857a476c7de9c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:45,054 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:45,054 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/B, priority=12, startTime=1680720824868; duration=0sec 2023-04-05 18:53:45,055 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:45,055 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:45,055 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:45,071 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45500 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:45,072 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/C is initiating minor compaction (all files) 2023-04-05 18:53:45,072 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/C in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:45,072 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3b741579c73d4381b3aa71013d10d592, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bdcb800ee1754159bf3f1a46f931af3b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/81b03c1c76de4ecfa2744ba71a9502a6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/276c7ec8623b48188af47a3173ed7a3f] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=44.4 K 2023-04-05 18:53:45,073 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 3b741579c73d4381b3aa71013d10d592, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680720821362 2023-04-05 18:53:45,074 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting bdcb800ee1754159bf3f1a46f931af3b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680720821799 2023-04-05 18:53:45,075 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 81b03c1c76de4ecfa2744ba71a9502a6, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680720822425 2023-04-05 18:53:45,076 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 276c7ec8623b48188af47a3173ed7a3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680720824213 2023-04-05 18:53:45,114 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#C#compaction#16 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:45,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720885133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720885133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720885139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720885140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720885141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720885337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720885338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720885345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720885346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720885346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,483 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/6a8f0ece9add4d73a6fe4721ff24fa3c 2023-04-05 18:53:45,549 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/97888120c78245409cdd29bc22f5a4ff as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/97888120c78245409cdd29bc22f5a4ff 2023-04-05 18:53:45,559 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/C of be0aa03376eb89131764dd34306ebfb5 into 97888120c78245409cdd29bc22f5a4ff(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:45,559 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:45,559 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/C, priority=12, startTime=1680720824868; duration=0sec 2023-04-05 18:53:45,559 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:45,560 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-05 18:53:45,561 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-05 18:53:45,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42912 deadline: 1680720885641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720885643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720885648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720885650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:45,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720885665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,924 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/7d819f334c8e46ae88e1a7efb3e8a865 2023-04-05 18:53:45,935 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/3163486607794186b204f79d2e1b51ba as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/3163486607794186b204f79d2e1b51ba 2023-04-05 18:53:45,947 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/3163486607794186b204f79d2e1b51ba, entries=150, sequenceid=289, filesize=11.9 K 2023-04-05 18:53:45,948 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/6a8f0ece9add4d73a6fe4721ff24fa3c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6a8f0ece9add4d73a6fe4721ff24fa3c 2023-04-05 18:53:45,971 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6a8f0ece9add4d73a6fe4721ff24fa3c, entries=150, sequenceid=289, filesize=11.9 K 2023-04-05 18:53:45,976 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/7d819f334c8e46ae88e1a7efb3e8a865 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7d819f334c8e46ae88e1a7efb3e8a865 2023-04-05 18:53:45,985 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7d819f334c8e46ae88e1a7efb3e8a865, entries=150, sequenceid=289, filesize=11.9 K 2023-04-05 18:53:45,988 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for be0aa03376eb89131764dd34306ebfb5 in 1067ms, sequenceid=289, compaction requested=false 2023-04-05 18:53:45,988 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:45,988 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:45,994 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:45,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:45,995 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:45,995 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:45,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:45,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb31773[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:45,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:45,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:45,999 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:45,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,000 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:46,000 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:46,000 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:46,000 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,000 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:46,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:46,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:46,010 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:46,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:46,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:46,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:46,010 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:46,014 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,014 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:46,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,018 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:46,020 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,022 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:46,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:46,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:46,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:46,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:46,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:46,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,028 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:46,028 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:46,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:46,133 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:46,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:46,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:46,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:46,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:46,139 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:46,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:46,140 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:46,140 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:46,141 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,141 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,143 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:46,144 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,145 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,145 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:46,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:46,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:46,148 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:46,148 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:46,154 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:46,154 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:46,155 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:53:46,155 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:46,155 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:46,155 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:46,155 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:46,156 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:46,156 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:46,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:46,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:46,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720886177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720886179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720886180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720886180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720886181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,219 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/5cb68cb5055f45f491d4ebcfb33afaa3 2023-04-05 18:53:46,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720886283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720886288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720886289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720886290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720886300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720886487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720886493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720886497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720886498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720886503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,656 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/715e5ce06f9d45af85f1e49e30bc97f7 2023-04-05 18:53:46,693 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/89f9f05466ab41dcbd901f662059fd11 2023-04-05 18:53:46,711 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/5cb68cb5055f45f491d4ebcfb33afaa3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5cb68cb5055f45f491d4ebcfb33afaa3 2023-04-05 18:53:46,720 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5cb68cb5055f45f491d4ebcfb33afaa3, entries=150, sequenceid=312, filesize=11.9 K 2023-04-05 18:53:46,722 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/715e5ce06f9d45af85f1e49e30bc97f7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/715e5ce06f9d45af85f1e49e30bc97f7 2023-04-05 18:53:46,729 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/715e5ce06f9d45af85f1e49e30bc97f7, entries=150, sequenceid=312, filesize=11.9 K 2023-04-05 18:53:46,731 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/89f9f05466ab41dcbd901f662059fd11 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/89f9f05466ab41dcbd901f662059fd11 2023-04-05 18:53:46,738 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/89f9f05466ab41dcbd901f662059fd11, entries=150, sequenceid=312, filesize=11.9 K 2023-04-05 18:53:46,740 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=73.80 KB/75570 for be0aa03376eb89131764dd34306ebfb5 in 584ms, sequenceid=312, compaction requested=true 2023-04-05 18:53:46,740 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:46,741 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:46,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,745 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:46,745 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,745 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:46,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:46,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e404b7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:46,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:46,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,749 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:46,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:46,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:46,750 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,750 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:46,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:46,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:46,762 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:46,762 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:46,762 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:46,762 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:46,762 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:46,764 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,765 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:46,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,765 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:46,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:46,765 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,769 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,772 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,772 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:46,772 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:46,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:46,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:46,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:46,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:46,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:46,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:46,774 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:46,774 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:46,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:46,793 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 18:53:46,793 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:46,793 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:46,793 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:46,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:46,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:46,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:46,822 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/5a30b03c7d6746789244cd16c6a0a6ae 2023-04-05 18:53:46,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720886826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720886831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720886836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720886837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720886838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,874 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:46,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:46,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:46,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:46,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:46,879 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:46,879 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:46,879 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:46,880 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:46,880 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,880 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,882 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:46,882 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:46,882 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,882 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:46,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:46,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:46,885 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:46,885 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:46,885 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:46,885 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:46,885 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:46,885 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:46,885 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:46,887 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:46,889 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,889 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:46,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:46,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7430a367[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:46,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:46,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:46,894 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,894 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:46,897 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:46,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:46,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:46,901 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:46,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:46,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:46,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:46,901 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:46,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:46,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:46,903 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:46,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:46,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:46,908 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,909 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,909 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:46,909 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:46,909 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:46,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:46,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:46,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:46,907 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:46,910 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:46,910 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:46,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:46,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720886939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720886939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720886943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:46,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720886943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:46,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720886943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,010 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:47,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:47,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:47,016 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:47,016 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,017 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:47,017 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:47,017 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,017 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,019 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,019 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,019 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,019 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:47,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:47,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:47,021 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:47,021 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:47,022 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,022 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:47,022 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:47,022 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,028 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:47,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f29f09[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:47,030 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:47,031 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,032 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:47,032 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,032 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:47,034 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,034 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:47,034 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:47,034 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:47,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:47,038 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:47,038 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:47,038 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:47,038 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:47,038 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:47,041 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,041 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:47,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,041 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:47,041 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:47,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,046 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:47,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:47,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:47,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:47,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:47,048 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:47,048 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:47,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720887144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720887144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720887145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720887145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,148 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:47,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720887146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:47,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:47,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:47,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:47,152 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:47,152 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:47,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:47,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,155 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:47,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:47,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:47,157 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:47,157 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:47,158 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,158 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:47,158 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:47,158 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:47,161 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,161 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,161 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,161 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:47,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13b912eb[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:47,163 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:47,163 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,164 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,164 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:47,164 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:47,164 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:47,165 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,165 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:47,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:47,166 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:47,169 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:47,169 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:47,169 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:47,170 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:47,170 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:47,172 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,172 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:47,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,172 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,173 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,178 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,178 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,178 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,178 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,178 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:47,178 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:47,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:47,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:47,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,178 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,179 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:47,179 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:47,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:47,179 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:47,179 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,179 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/1519d9f1feb2450daa80ba868df60901 2023-04-05 18:53:47,279 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:47,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:47,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:47,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:47,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:47,285 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:47,285 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:47,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:47,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,287 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:47,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:47,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:47,289 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:47,289 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:47,290 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,290 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:47,290 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:47,290 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:47,292 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,292 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,292 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,292 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:47,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167d6d97[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:47,294 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:47,294 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,296 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,296 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:47,296 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,296 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:47,296 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:47,298 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:47,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:47,302 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:47,302 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:47,302 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:47,302 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:47,302 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:47,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:47,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:47,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:47,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,322 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:47,323 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:47,323 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:47,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:47,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,423 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:47,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:47,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:47,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:47,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:47,428 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:47,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:47,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:47,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:47,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:47,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:47,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:47,435 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,435 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:47,435 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:47,435 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:47,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:47,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6727dfa4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:47,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:47,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:47,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:47,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:47,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:47,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:47,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720887448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720887449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:47,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:47,450 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:47,450 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:47,450 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:47,450 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:47,450 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:47,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:47,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:47,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720887450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720887449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,453 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720887451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,455 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:47,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:47,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:47,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,464 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:47,464 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:47,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:47,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:47,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:47,564 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:47,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:47,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:47,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:47,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:47,568 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:47,569 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:47,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:47,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,572 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:47,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:47,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:47,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:47,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:47,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:47,576 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,576 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:47,576 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:47,576 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:47,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:47,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23e90e9e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:47,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:47,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:47,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:47,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:47,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:47,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:47,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:47,599 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:47,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:47,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:47,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:47,599 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:47,601 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,601 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,601 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:47,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,603 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,605 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:47,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:47,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:47,610 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,610 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:47,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:47,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:47,610 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:47,610 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:47,682 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/7ef422707c214a228e1ec9ac731e08ab 2023-04-05 18:53:47,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/5a30b03c7d6746789244cd16c6a0a6ae as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5a30b03c7d6746789244cd16c6a0a6ae 2023-04-05 18:53:47,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5a30b03c7d6746789244cd16c6a0a6ae, entries=150, sequenceid=327, filesize=11.9 K 2023-04-05 18:53:47,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/1519d9f1feb2450daa80ba868df60901 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1519d9f1feb2450daa80ba868df60901 2023-04-05 18:53:47,708 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1519d9f1feb2450daa80ba868df60901, entries=150, sequenceid=327, filesize=11.9 K 2023-04-05 18:53:47,709 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/7ef422707c214a228e1ec9ac731e08ab as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ef422707c214a228e1ec9ac731e08ab 2023-04-05 18:53:47,710 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:47,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:47,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:47,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:47,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:47,716 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:47,717 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,720 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:47,721 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:47,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ef422707c214a228e1ec9ac731e08ab, entries=150, sequenceid=327, filesize=11.9 K 2023-04-05 18:53:47,721 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,721 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,723 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for be0aa03376eb89131764dd34306ebfb5 in 930ms, sequenceid=327, compaction requested=true 2023-04-05 18:53:47,723 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:47,723 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:47,723 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:47,723 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:47,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:47,723 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:47,723 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:47,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:47,724 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,724 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:47,725 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:47,726 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/A is initiating minor compaction (all files) 2023-04-05 18:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:47,726 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/A in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,726 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:47,726 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/490a41f41cf646a48cd087ac64387c56, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/3163486607794186b204f79d2e1b51ba, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5cb68cb5055f45f491d4ebcfb33afaa3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5a30b03c7d6746789244cd16c6a0a6ae] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=47.7 K 2023-04-05 18:53:47,726 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/B is initiating minor compaction (all files) 2023-04-05 18:53:47,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:47,727 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/B in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,727 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4b2c981f2fba455a9a6857a476c7de9c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6a8f0ece9add4d73a6fe4721ff24fa3c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/715e5ce06f9d45af85f1e49e30bc97f7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1519d9f1feb2450daa80ba868df60901] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=47.7 K 2023-04-05 18:53:47,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:47,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:47,727 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:47,727 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 490a41f41cf646a48cd087ac64387c56, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680720824213 2023-04-05 18:53:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:47,729 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:47,729 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:47,729 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:47,729 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,729 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:47,729 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-05 18:53:47,730 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:47,730 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:47,730 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:47,730 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:47,730 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:47,730 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:47,733 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4b2c981f2fba455a9a6857a476c7de9c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680720824213 2023-04-05 18:53:47,734 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 3163486607794186b204f79d2e1b51ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680720824291 2023-04-05 18:53:47,734 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 6a8f0ece9add4d73a6fe4721ff24fa3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680720824291 2023-04-05 18:53:47,734 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 5cb68cb5055f45f491d4ebcfb33afaa3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680720825010 2023-04-05 18:53:47,735 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 715e5ce06f9d45af85f1e49e30bc97f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680720825010 2023-04-05 18:53:47,735 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 5a30b03c7d6746789244cd16c6a0a6ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680720826177 2023-04-05 18:53:47,736 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1519d9f1feb2450daa80ba868df60901, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680720826177 2023-04-05 18:53:47,769 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#A#compaction#25 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:47,770 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#B#compaction#26 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:47,817 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/38650f87ad75468e93b3bbc95999d3eb as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/38650f87ad75468e93b3bbc95999d3eb 2023-04-05 18:53:47,827 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/a737367978d044878d16041b6407be23 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a737367978d044878d16041b6407be23 2023-04-05 18:53:47,831 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/A of be0aa03376eb89131764dd34306ebfb5 into 38650f87ad75468e93b3bbc95999d3eb(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:47,832 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:47,832 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/A, priority=12, startTime=1680720827723; duration=0sec 2023-04-05 18:53:47,832 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:47,832 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:47,837 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:47,837 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/C is initiating minor compaction (all files) 2023-04-05 18:53:47,837 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/C in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:47,837 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/97888120c78245409cdd29bc22f5a4ff, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7d819f334c8e46ae88e1a7efb3e8a865, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/89f9f05466ab41dcbd901f662059fd11, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ef422707c214a228e1ec9ac731e08ab] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=47.7 K 2023-04-05 18:53:47,839 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 97888120c78245409cdd29bc22f5a4ff, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680720824213 2023-04-05 18:53:47,845 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/B of be0aa03376eb89131764dd34306ebfb5 into a737367978d044878d16041b6407be23(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:47,846 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:47,846 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/B, priority=12, startTime=1680720827723; duration=0sec 2023-04-05 18:53:47,846 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:47,846 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7d819f334c8e46ae88e1a7efb3e8a865, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680720824291 2023-04-05 18:53:47,847 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 89f9f05466ab41dcbd901f662059fd11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680720825010 2023-04-05 18:53:47,850 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7ef422707c214a228e1ec9ac731e08ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680720826177 2023-04-05 18:53:47,868 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#C#compaction#27 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:47,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:47,952 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:47,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4442) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4381) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4857) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4851) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4847) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3159) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720887964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720887967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720887969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720887968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:47,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:47,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720887969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720888070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720888073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720888074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720888074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720888075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,194 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/b4e7a6cbdd8440b882de174703cec221 2023-04-05 18:53:48,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720888277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720888278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720888278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720888278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720888278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,294 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/a4f524bb1e95416a902128a9f0d8e25b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a4f524bb1e95416a902128a9f0d8e25b 2023-04-05 18:53:48,306 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/C of be0aa03376eb89131764dd34306ebfb5 into a4f524bb1e95416a902128a9f0d8e25b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:48,306 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:48,306 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/C, priority=12, startTime=1680720827724; duration=0sec 2023-04-05 18:53:48,306 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:48,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720888581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720888582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720888582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720888582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:48,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720888585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,617 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/31a6aea2a6714c1f9570c2322c55b55b 2023-04-05 18:53:48,640 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/3eccb99c5ff94614bf7fa6695119db85 2023-04-05 18:53:48,647 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/b4e7a6cbdd8440b882de174703cec221 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/b4e7a6cbdd8440b882de174703cec221 2023-04-05 18:53:48,657 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/b4e7a6cbdd8440b882de174703cec221, entries=150, sequenceid=344, filesize=11.9 K 2023-04-05 18:53:48,658 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/31a6aea2a6714c1f9570c2322c55b55b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/31a6aea2a6714c1f9570c2322c55b55b 2023-04-05 18:53:48,669 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/31a6aea2a6714c1f9570c2322c55b55b, entries=150, sequenceid=344, filesize=11.9 K 2023-04-05 18:53:48,671 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/3eccb99c5ff94614bf7fa6695119db85 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3eccb99c5ff94614bf7fa6695119db85 2023-04-05 18:53:48,681 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3eccb99c5ff94614bf7fa6695119db85, entries=150, sequenceid=344, filesize=11.9 K 2023-04-05 18:53:48,685 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for be0aa03376eb89131764dd34306ebfb5 in 956ms, sequenceid=344, compaction requested=false 2023-04-05 18:53:48,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:48,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:48,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:48,688 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:48,688 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@781c7e42[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:48,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:48,691 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:48,692 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:48,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:48,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:48,695 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,695 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:48,695 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:48,695 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:48,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:48,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:48,701 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:48,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:48,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:48,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:48,701 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:48,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:48,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:48,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:48,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:48,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:48,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:48,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:48,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:48,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:48,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:48,711 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,711 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:48,711 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:48,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:48,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:48,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:48,711 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:48,720 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:48,720 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:48,720 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:48,721 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:48,721 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:48,821 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:48,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:48,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:48,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:48,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:48,825 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:48,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:48,826 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:48,826 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:48,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:48,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:48,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:48,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:48,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:48,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:48,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:48,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:48,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:48,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:48,837 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:48,838 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:48,838 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:48,838 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:48,838 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 18:53:48,838 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:48,838 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:48,838 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:48,838 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:48,838 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:48,838 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:48,853 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/c2abdd42e49b42e3b74a738f812202fc 2023-04-05 18:53:48,895 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/5c975630cb554c73ba96a91d1a3a2ac9 2023-04-05 18:53:48,940 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/26858bd6beeb4938b49dbe03abecd25a 2023-04-05 18:53:48,960 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/c2abdd42e49b42e3b74a738f812202fc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/c2abdd42e49b42e3b74a738f812202fc 2023-04-05 18:53:48,969 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/c2abdd42e49b42e3b74a738f812202fc, entries=150, sequenceid=362, filesize=11.9 K 2023-04-05 18:53:48,972 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/5c975630cb554c73ba96a91d1a3a2ac9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5c975630cb554c73ba96a91d1a3a2ac9 2023-04-05 18:53:48,987 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5c975630cb554c73ba96a91d1a3a2ac9, entries=150, sequenceid=362, filesize=11.9 K 2023-04-05 18:53:48,990 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/26858bd6beeb4938b49dbe03abecd25a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/26858bd6beeb4938b49dbe03abecd25a 2023-04-05 18:53:48,999 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/26858bd6beeb4938b49dbe03abecd25a, entries=150, sequenceid=362, filesize=11.9 K 2023-04-05 18:53:49,002 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for be0aa03376eb89131764dd34306ebfb5 in 164ms, sequenceid=362, compaction requested=true 2023-04-05 18:53:49,002 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:49,002 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:49,006 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:49,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f0b503[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:49,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:49,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,011 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,012 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:49,012 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:49,012 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:49,012 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,012 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:49,014 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:49,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:49,026 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:49,026 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:49,026 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:49,026 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:49,026 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:49,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,032 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,035 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,042 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,042 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,042 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,042 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,042 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,042 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,042 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,042 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,042 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:49,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:49,045 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:49,045 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:49,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:49,095 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:53:49,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:49,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:49,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:49,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:49,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:49,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:49,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680720829102, max timestamp=1680720829121Num uniques 859; 2023-04-05 18:53:49,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680720829102, max timestamp=1680720829121Num uniques 859; 2023-04-05 18:53:49,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680720829102, max timestamp=1680720829121Num uniques 859; 2023-04-05 18:53:49,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 18:53:49,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 18:53:49,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 18:53:49,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/0a281cdc445d4aba9d7dfbaa5aa2de03 2023-04-05 18:53:49,145 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:49,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:49,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:49,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:49,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:49,179 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:49,179 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:49,183 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:49,183 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:49,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,186 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:49,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720889176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720889169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:49,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720889189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720889189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720889190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:49,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:49,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:49,213 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:49,220 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,220 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:49,220 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:49,220 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:49,221 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:49,221 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,221 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:49,221 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:49,228 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,228 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,229 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:49,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c7380e6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:49,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:49,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:49,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:49,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:49,235 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,235 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:49,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:49,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:49,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:49,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:49,246 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:49,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:49,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:49,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:49,247 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:49,249 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,249 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,249 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,251 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,252 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,256 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,256 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,256 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,256 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:49,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:49,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,257 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:49,257 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:49,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:49,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720889292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720889294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720889303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720889303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720889304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,357 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:49,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:49,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:49,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:49,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:49,363 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:49,363 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:49,365 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:49,365 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:49,365 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,366 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,367 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:49,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:49,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:49,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:49,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:49,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:49,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:49,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:49,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:49,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:49,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:49,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:49,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:49,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3705f808[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:49,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,379 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:49,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:49,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:49,379 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,379 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:49,383 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:49,388 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:49,388 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:49,388 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:49,388 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:49,388 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:49,390 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,390 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,390 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,394 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,397 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:49,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:49,401 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:49,401 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:49,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,501 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:49,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720889500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:49,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:49,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:49,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:49,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,508 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:49,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720889502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:49,509 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:49,509 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:49,509 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,509 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720889508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720889513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:49,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720889516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:49,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:49,532 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,532 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:49,533 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:49,533 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:49,539 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,539 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,540 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,540 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:49,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c707ec4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:49,542 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:49,542 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,544 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,544 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:49,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:49,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:49,545 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:49,549 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:49,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:49,552 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:49,552 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:49,552 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:49,552 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:49,552 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:49,554 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,554 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,554 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,555 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,557 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,565 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:49,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:49,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,565 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,565 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:49,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:49,565 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:49,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,628 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/193826160b284e3ca71efc82fe04d131 2023-04-05 18:53:49,665 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:49,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:49,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:49,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:49,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:49,673 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:49,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:49,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:49,674 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:49,674 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,674 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,675 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,676 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,677 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,677 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:49,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:49,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:49,695 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:49,695 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:49,696 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,696 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:49,696 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:49,696 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:49,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:49,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c289ee4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:49,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:49,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,712 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:49,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:49,721 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,721 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:49,721 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,721 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:49,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:49,729 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:49,729 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:49,729 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:49,729 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:49,729 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:49,731 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,732 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,732 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,733 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,736 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,739 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:49,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:49,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,740 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,740 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,740 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,740 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,740 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:49,740 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:49,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:49,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720889806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720889813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720889817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42904 deadline: 1680720889817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:49,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42958 deadline: 1680720889823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,840 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:49,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:49,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:49,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:49,848 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:49,848 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:49,848 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:49,848 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:49,848 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,850 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,850 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,851 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,851 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:49,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:49,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:49,856 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:49,856 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:49,856 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:49,856 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:49,856 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:49,856 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,856 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:49,857 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:49,857 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:49,861 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:49,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c1c4f37[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:49,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:49,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,870 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:49,870 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:49,870 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:49,871 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,871 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:49,872 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:49,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:49,876 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:49,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:49,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:49,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:49,876 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:49,878 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,878 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,878 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:49,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:49,881 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:49,883 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,887 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:49,887 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:49,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:49,887 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,888 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:49,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:49,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:49,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:49,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:49,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:49,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:49,888 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:49,888 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:49,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:49,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:49,988 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:50,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:50,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:50,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:50,002 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:50,002 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,002 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:50,002 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:50,003 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,003 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,004 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,004 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,004 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,004 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:50,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:50,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:50,011 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,011 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:50,011 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:50,011 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,012 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:50,012 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:50,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:50,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c12f04[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:50,016 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:50,016 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,017 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:50,018 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,018 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:50,021 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:50,024 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:50,024 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:50,024 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:50,024 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:50,024 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:50,026 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,026 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,026 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,030 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,037 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,037 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,037 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,037 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,037 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,037 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,038 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,038 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,038 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,038 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:50,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:50,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,039 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:50,039 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:50,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/6aac9f7f11f54724b0bc2c7bb0de60ae 2023-04-05 18:53:50,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/0a281cdc445d4aba9d7dfbaa5aa2de03 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/0a281cdc445d4aba9d7dfbaa5aa2de03 2023-04-05 18:53:50,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/0a281cdc445d4aba9d7dfbaa5aa2de03, entries=150, sequenceid=372, filesize=11.9 K 2023-04-05 18:53:50,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/193826160b284e3ca71efc82fe04d131 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/193826160b284e3ca71efc82fe04d131 2023-04-05 18:53:50,136 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/193826160b284e3ca71efc82fe04d131, entries=150, sequenceid=372, filesize=11.9 K 2023-04-05 18:53:50,137 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/6aac9f7f11f54724b0bc2c7bb0de60ae as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6aac9f7f11f54724b0bc2c7bb0de60ae 2023-04-05 18:53:50,139 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:50,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:50,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:50,149 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6aac9f7f11f54724b0bc2c7bb0de60ae, entries=150, sequenceid=372, filesize=11.9 K 2023-04-05 18:53:50,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for be0aa03376eb89131764dd34306ebfb5 in 1055ms, sequenceid=372, compaction requested=true 2023-04-05 18:53:50,150 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:50,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:50,150 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:50,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:50,150 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:50,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:50,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:50,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:50,152 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:50,152 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,152 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:50,152 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:50,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,153 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:50,153 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/A is initiating minor compaction (all files) 2023-04-05 18:53:50,154 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/A in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,154 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/38650f87ad75468e93b3bbc95999d3eb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/b4e7a6cbdd8440b882de174703cec221, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/c2abdd42e49b42e3b74a738f812202fc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/0a281cdc445d4aba9d7dfbaa5aa2de03] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=47.9 K 2023-04-05 18:53:50,154 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:50,154 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/B is initiating minor compaction (all files) 2023-04-05 18:53:50,154 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/B in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,154 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a737367978d044878d16041b6407be23, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/31a6aea2a6714c1f9570c2322c55b55b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5c975630cb554c73ba96a91d1a3a2ac9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/193826160b284e3ca71efc82fe04d131] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=47.9 K 2023-04-05 18:53:50,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,157 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 38650f87ad75468e93b3bbc95999d3eb, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680720826177 2023-04-05 18:53:50,157 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a737367978d044878d16041b6407be23, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680720826177 2023-04-05 18:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,157 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting b4e7a6cbdd8440b882de174703cec221, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1680720826830 2023-04-05 18:53:50,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,158 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,158 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:50,158 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 31a6aea2a6714c1f9570c2322c55b55b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1680720826830 2023-04-05 18:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:50,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:50,159 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c2abdd42e49b42e3b74a738f812202fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680720827966 2023-04-05 18:53:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:50,159 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,159 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:50,159 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-05 18:53:50,160 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 5c975630cb554c73ba96a91d1a3a2ac9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680720827966 2023-04-05 18:53:50,160 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:50,160 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 0a281cdc445d4aba9d7dfbaa5aa2de03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680720829088 2023-04-05 18:53:50,160 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:50,160 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:50,160 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:50,160 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:50,160 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:50,160 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 193826160b284e3ca71efc82fe04d131, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680720829088 2023-04-05 18:53:50,177 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#A#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:50,177 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#B#compaction#38 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:50,191 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/db31589c9b9f4527aad8f88a4cada90d 2023-04-05 18:53:50,213 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/0cc30d20932f4f72ac10181e867ebf6d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0cc30d20932f4f72ac10181e867ebf6d 2023-04-05 18:53:50,214 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/7f15aae80622403e8010ba5f6a331745 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/7f15aae80622403e8010ba5f6a331745 2023-04-05 18:53:50,227 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/A of be0aa03376eb89131764dd34306ebfb5 into 7f15aae80622403e8010ba5f6a331745(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:50,227 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:50,227 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/A, priority=12, startTime=1680720830150; duration=0sec 2023-04-05 18:53:50,229 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/B of be0aa03376eb89131764dd34306ebfb5 into 0cc30d20932f4f72ac10181e867ebf6d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:50,229 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:50,229 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/B, priority=12, startTime=1680720830150; duration=0sec 2023-04-05 18:53:50,230 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:50,230 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:50,230 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:50,235 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:50,235 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/C is initiating minor compaction (all files) 2023-04-05 18:53:50,235 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/C in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,235 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a4f524bb1e95416a902128a9f0d8e25b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3eccb99c5ff94614bf7fa6695119db85, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/26858bd6beeb4938b49dbe03abecd25a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6aac9f7f11f54724b0bc2c7bb0de60ae] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=47.9 K 2023-04-05 18:53:50,236 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting a4f524bb1e95416a902128a9f0d8e25b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680720826177 2023-04-05 18:53:50,236 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 3eccb99c5ff94614bf7fa6695119db85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1680720826830 2023-04-05 18:53:50,237 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 26858bd6beeb4938b49dbe03abecd25a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680720827966 2023-04-05 18:53:50,237 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6aac9f7f11f54724b0bc2c7bb0de60ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680720829088 2023-04-05 18:53:50,244 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/7fc30bdaceb54a82a95cef1cfddd790a 2023-04-05 18:53:50,250 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#C#compaction#40 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:50,280 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/0bc8f03c3eed45a19fb5450c7dd147bc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/0bc8f03c3eed45a19fb5450c7dd147bc 2023-04-05 18:53:50,288 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/C of be0aa03376eb89131764dd34306ebfb5 into 0bc8f03c3eed45a19fb5450c7dd147bc(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:50,288 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:50,288 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/C, priority=12, startTime=1680720830151; duration=0sec 2023-04-05 18:53:50,288 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:50,292 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/03113f63815046188034c20774a11760 2023-04-05 18:53:50,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/db31589c9b9f4527aad8f88a4cada90d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/db31589c9b9f4527aad8f88a4cada90d 2023-04-05 18:53:50,309 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/db31589c9b9f4527aad8f88a4cada90d, entries=150, sequenceid=407, filesize=11.9 K 2023-04-05 18:53:50,310 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/7fc30bdaceb54a82a95cef1cfddd790a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/7fc30bdaceb54a82a95cef1cfddd790a 2023-04-05 18:53:50,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:50,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:50,328 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/7fc30bdaceb54a82a95cef1cfddd790a, entries=150, sequenceid=407, filesize=11.9 K 2023-04-05 18:53:50,329 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/03113f63815046188034c20774a11760 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/03113f63815046188034c20774a11760 2023-04-05 18:53:50,336 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/03113f63815046188034c20774a11760, entries=150, sequenceid=407, filesize=11.9 K 2023-04-05 18:53:50,337 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=33.54 KB/34350 for be0aa03376eb89131764dd34306ebfb5 in 178ms, sequenceid=407, compaction requested=false 2023-04-05 18:53:50,337 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:50,337 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,337 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:50,343 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,343 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,343 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,343 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b68e906[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:50,347 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:50,347 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,347 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:53:50,348 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:50,348 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:50,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:50,348 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:50,349 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:50,349 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:50,349 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:50,349 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,349 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:50,349 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:50,349 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:50,349 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,349 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:50,353 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:50,354 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,354 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,354 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:50,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:50,360 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:50,360 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:50,360 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:50,360 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:50,360 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:50,364 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,364 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,364 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,365 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720830365, max timestamp=1680720830377Num uniques 859; 2023-04-05 18:53:50,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:53:50,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720830365, max timestamp=1680720830377Num uniques 859; 2023-04-05 18:53:50,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:53:50,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720830365, max timestamp=1680720830377Num uniques 859; 2023-04-05 18:53:50,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:53:50,383 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,383 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,383 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,383 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,383 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,383 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,383 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,383 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,383 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,384 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:50,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:50,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,384 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:50,384 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/2b1b8abcd52f448c81cf66fe79ef4406 2023-04-05 18:53:50,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720890414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720890418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720890420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42922 deadline: 1680720890422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720890423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/1c01290c378c4644b1dab9d22982d3b2 2023-04-05 18:53:50,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/847802b9cb4b4af6845304c323bb862a 2023-04-05 18:53:50,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/2b1b8abcd52f448c81cf66fe79ef4406 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/2b1b8abcd52f448c81cf66fe79ef4406 2023-04-05 18:53:50,484 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:50,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:50,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:50,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/2b1b8abcd52f448c81cf66fe79ef4406, entries=100, sequenceid=420, filesize=9.5 K 2023-04-05 18:53:50,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:50,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:50,489 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:50,489 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,489 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:50,489 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:50,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/1c01290c378c4644b1dab9d22982d3b2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1c01290c378c4644b1dab9d22982d3b2 2023-04-05 18:53:50,490 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,490 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,491 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,491 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,492 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,492 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:50,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:50,493 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:50,493 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,493 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:50,493 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:50,494 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:50,494 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,494 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:50,494 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:50,494 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,494 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:50,496 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,496 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,497 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,497 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1c01290c378c4644b1dab9d22982d3b2, entries=100, sequenceid=420, filesize=9.5 K 2023-04-05 18:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/847802b9cb4b4af6845304c323bb862a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/847802b9cb4b4af6845304c323bb862a 2023-04-05 18:53:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:50,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@335f9331[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:50,500 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:50,500 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,503 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:50,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:50,508 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/847802b9cb4b4af6845304c323bb862a, entries=100, sequenceid=420, filesize=9.5 K 2023-04-05 18:53:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for be0aa03376eb89131764dd34306ebfb5 in 164ms, sequenceid=420, compaction requested=true 2023-04-05 18:53:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:50,512 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:50,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:50,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:50,512 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:50,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:50,514 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:50,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:50,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:50,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:50,514 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:50,515 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34524 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:50,515 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/A is initiating minor compaction (all files) 2023-04-05 18:53:50,515 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/A in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,515 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/7f15aae80622403e8010ba5f6a331745, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/db31589c9b9f4527aad8f88a4cada90d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/2b1b8abcd52f448c81cf66fe79ef4406] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=33.7 K 2023-04-05 18:53:50,516 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7f15aae80622403e8010ba5f6a331745, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680720829088 2023-04-05 18:53:50,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,520 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34524 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:50,521 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/B is initiating minor compaction (all files) 2023-04-05 18:53:50,521 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/B in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,521 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0cc30d20932f4f72ac10181e867ebf6d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/7fc30bdaceb54a82a95cef1cfddd790a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1c01290c378c4644b1dab9d22982d3b2] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=33.7 K 2023-04-05 18:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,521 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,521 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,522 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting db31589c9b9f4527aad8f88a4cada90d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680720829145 2023-04-05 18:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,523 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 2b1b8abcd52f448c81cf66fe79ef4406, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680720830324 2023-04-05 18:53:50,523 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0cc30d20932f4f72ac10181e867ebf6d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680720829088 2023-04-05 18:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,524 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 7fc30bdaceb54a82a95cef1cfddd790a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680720829145 2023-04-05 18:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,525 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1c01290c378c4644b1dab9d22982d3b2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680720830324 2023-04-05 18:53:50,525 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,529 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,529 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,529 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,530 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,530 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,530 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,530 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,530 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,530 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,530 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:50,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:50,530 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:50,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:50,530 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:50,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-05 18:53:50,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:50,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:50,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:53:50,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:50,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:53:50,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:50,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:53:50,540 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#B#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:50,541 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#A#compaction#47 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:50,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720890535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720890535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720890537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720890540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720890541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/14f6cd9d94654480b0aecc56aef769a5 2023-04-05 18:53:50,596 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/210e6e99220f4dfc8b12438a91e13cdc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/210e6e99220f4dfc8b12438a91e13cdc 2023-04-05 18:53:50,603 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/B of be0aa03376eb89131764dd34306ebfb5 into 210e6e99220f4dfc8b12438a91e13cdc(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:50,603 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:50,603 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/B, priority=13, startTime=1680720830512; duration=0sec 2023-04-05 18:53:50,604 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:50,604 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:50,605 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34524 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:50,605 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/C is initiating minor compaction (all files) 2023-04-05 18:53:50,605 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/C in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,605 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/0bc8f03c3eed45a19fb5450c7dd147bc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/03113f63815046188034c20774a11760, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/847802b9cb4b4af6845304c323bb862a] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=33.7 K 2023-04-05 18:53:50,606 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0bc8f03c3eed45a19fb5450c7dd147bc, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680720829088 2023-04-05 18:53:50,606 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 03113f63815046188034c20774a11760, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680720829145 2023-04-05 18:53:50,607 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 847802b9cb4b4af6845304c323bb862a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680720830324 2023-04-05 18:53:50,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/243eb79f795b403a9e4f0771ecd9642c 2023-04-05 18:53:50,617 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#C#compaction#49 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:50,631 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:50,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:50,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:50,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720890644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720890644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720890645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42946 deadline: 1680720890646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:50,654 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:50,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:50,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:50,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,657 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,658 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,658 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,659 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:50,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:50,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:50,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/7ba937be173045d1a83c93fc11ce7f52 2023-04-05 18:53:50,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:50,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:50,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:50,667 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/b3dbd554bf77452fbbacdbeb5d7706d2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/b3dbd554bf77452fbbacdbeb5d7706d2 2023-04-05 18:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:50,672 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,672 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:50,672 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:50,672 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:50,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/14f6cd9d94654480b0aecc56aef769a5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/14f6cd9d94654480b0aecc56aef769a5 2023-04-05 18:53:50,674 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,674 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/C of be0aa03376eb89131764dd34306ebfb5 into b3dbd554bf77452fbbacdbeb5d7706d2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,674 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:50,674 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,674 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/C, priority=13, startTime=1680720830512; duration=0sec 2023-04-05 18:53:50,675 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/14f6cd9d94654480b0aecc56aef769a5, entries=150, sequenceid=459, filesize=11.9 K 2023-04-05 18:53:50,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/243eb79f795b403a9e4f0771ecd9642c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/243eb79f795b403a9e4f0771ecd9642c 2023-04-05 18:53:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:50,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d34079a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:50,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:50,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,683 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:50,684 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,684 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:50,685 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:50,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/243eb79f795b403a9e4f0771ecd9642c, entries=150, sequenceid=459, filesize=11.9 K 2023-04-05 18:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/7ba937be173045d1a83c93fc11ce7f52 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ba937be173045d1a83c93fc11ce7f52 2023-04-05 18:53:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:50,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:50,693 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:50,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:50,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:50,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:50,693 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:50,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,697 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,702 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,702 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,702 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:50,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:50,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:50,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,703 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,703 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:50,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ba937be173045d1a83c93fc11ce7f52, entries=150, sequenceid=459, filesize=11.9 K 2023-04-05 18:53:50,706 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~440.72 KB/451296, currentSize=26.84 KB/27480 for be0aa03376eb89131764dd34306ebfb5 in 175ms, sequenceid=459, compaction requested=false 2023-04-05 18:53:50,706 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:50,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:50,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:53:50,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:50,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:50,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:50,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:50,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:50,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:50,784 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/393e15aefed34766aba24d1cbb24e6c3 2023-04-05 18:53:50,804 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:50,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:50,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:50,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:50,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:50,808 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:50,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:50,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:50,809 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,809 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,811 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,811 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:50,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:50,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:50,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:50,813 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,813 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:50,813 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:50,813 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:50,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:50,819 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:50,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:50,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4070b637[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:50,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,823 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:50,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720830810, max timestamp=1680720830821Num uniques 859; 2023-04-05 18:53:50,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:53:50,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720830810, max timestamp=1680720830821Num uniques 859; 2023-04-05 18:53:50,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:53:50,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:50,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:50,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:50,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720830810, max timestamp=1680720830821Num uniques 859; 2023-04-05 18:53:50,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:53:50,832 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:50,840 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:50,840 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:50,840 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:50,840 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:50,840 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:50,843 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,843 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,843 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,844 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,846 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,849 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,849 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,849 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:50,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:50,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,849 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:50,849 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:50,849 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,849 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,849 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,850 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,850 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,850 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720890860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720890860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680720830855, max timestamp=1680720830864Num uniques 1018; 2023-04-05 18:53:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-05 18:53:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680720830855, max timestamp=1680720830864Num uniques 1018; 2023-04-05 18:53:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680720830855, max timestamp=1680720830864Num uniques 1018; 2023-04-05 18:53:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-05 18:53:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-05 18:53:50,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42946 deadline: 1680720890887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720890892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720890900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,949 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:50,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:50,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:50,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:50,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:50,957 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:50,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:50,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:50,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,960 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:50,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:50,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:50,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:50,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:50,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:50,964 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,964 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:50,965 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:50,965 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:50,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:50,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:50,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:50,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:50,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:50,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720890969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:50,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720890969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:50,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ccc57e6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:50,983 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:50,983 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,983 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/344b59a558f14d55af661384ca05350b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/344b59a558f14d55af661384ca05350b 2023-04-05 18:53:50,984 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:50,985 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:50,985 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:50,985 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,985 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:50,985 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:50,988 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:50,988 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,993 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/A of be0aa03376eb89131764dd34306ebfb5 into 344b59a558f14d55af661384ca05350b(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:50,993 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:50,993 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/A, priority=13, startTime=1680720830512; duration=0sec 2023-04-05 18:53:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,993 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:50,994 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:50,994 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:50,994 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:50,994 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:50,994 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:50,996 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,996 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,996 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:50,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:50,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:50,999 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42946 deadline: 1680720890997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720890998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720891002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,010 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,011 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,011 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:51,011 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:51,011 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,011 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,011 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,011 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:51,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:51,011 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:51,012 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:51,012 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:51,011 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:51,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:51,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:51,112 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:51,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:51,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:51,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:51,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:51,118 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:51,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:51,119 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:51,119 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:51,119 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,119 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,122 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:51,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:51,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:51,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:51,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:51,128 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:51,128 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:51,128 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:51,128 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:51,129 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:51,129 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:51,131 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,131 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c58155[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:51,135 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:51,135 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,137 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:51,139 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:51,139 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:51,139 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:51,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,140 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:51,140 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:51,140 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:51,140 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:51,140 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:51,140 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:51,140 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:51,142 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,142 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,142 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:51,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,149 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,150 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:51,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:51,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:51,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:51,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,154 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:51,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:51,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:51,154 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:51,154 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:51,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720891174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720891175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42946 deadline: 1680720891203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720891203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720891206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/4d9a36b7ac124288966c02af96cb9204 2023-04-05 18:53:51,254 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:51,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:51,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:51,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:51,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:51,276 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:51,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:51,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:51,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:51,277 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,277 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:51,279 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,280 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,280 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:51,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:51,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:51,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:51,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/a46c44e642a5447a96f87d9c53761d09 2023-04-05 18:53:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:51,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/393e15aefed34766aba24d1cbb24e6c3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/393e15aefed34766aba24d1cbb24e6c3 2023-04-05 18:53:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:51,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:51,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:51,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:51,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:51,303 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:51,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:51,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e504c60[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:51,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:51,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,309 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:51,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:51,312 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/393e15aefed34766aba24d1cbb24e6c3, entries=200, sequenceid=471, filesize=14.3 K 2023-04-05 18:53:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:51,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:51,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/4d9a36b7ac124288966c02af96cb9204 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4d9a36b7ac124288966c02af96cb9204 2023-04-05 18:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4d9a36b7ac124288966c02af96cb9204, entries=150, sequenceid=471, filesize=11.9 K 2023-04-05 18:53:51,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/a46c44e642a5447a96f87d9c53761d09 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a46c44e642a5447a96f87d9c53761d09 2023-04-05 18:53:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:51,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:51,324 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:51,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:51,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:51,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:51,324 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:51,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:51,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:51,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:51,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:51,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:51,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:51,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:51,334 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:51,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:51,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,335 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:51,335 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:51,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:51,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:51,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a46c44e642a5447a96f87d9c53761d09, entries=150, sequenceid=471, filesize=11.9 K 2023-04-05 18:53:51,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for be0aa03376eb89131764dd34306ebfb5 in 589ms, sequenceid=471, compaction requested=true 2023-04-05 18:53:51,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:51,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:51,348 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:51,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:51,348 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:51,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:51,349 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:51,350 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/A is initiating minor compaction (all files) 2023-04-05 18:53:51,350 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/A in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:51,350 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/344b59a558f14d55af661384ca05350b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/14f6cd9d94654480b0aecc56aef769a5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/393e15aefed34766aba24d1cbb24e6c3] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=38.6 K 2023-04-05 18:53:51,350 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 344b59a558f14d55af661384ca05350b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680720829176 2023-04-05 18:53:51,355 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:51,356 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/B is initiating minor compaction (all files) 2023-04-05 18:53:51,356 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/B in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:51,356 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/210e6e99220f4dfc8b12438a91e13cdc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/243eb79f795b403a9e4f0771ecd9642c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4d9a36b7ac124288966c02af96cb9204] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=36.2 K 2023-04-05 18:53:51,355 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 14f6cd9d94654480b0aecc56aef769a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680720830419 2023-04-05 18:53:51,357 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 210e6e99220f4dfc8b12438a91e13cdc, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680720829176 2023-04-05 18:53:51,359 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 393e15aefed34766aba24d1cbb24e6c3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680720830533 2023-04-05 18:53:51,359 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 243eb79f795b403a9e4f0771ecd9642c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680720830419 2023-04-05 18:53:51,360 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4d9a36b7ac124288966c02af96cb9204, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680720830533 2023-04-05 18:53:51,381 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#A#compaction#54 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:51,393 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#B#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:51,435 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:51,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:51,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:51,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:51,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:51,440 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:51,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:51,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:51,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:51,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:51,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:51,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:51,447 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,447 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:51,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:51,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:51,448 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/2ea9de25f9474eb694736ea7297adf45 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2ea9de25f9474eb694736ea7297adf45 2023-04-05 18:53:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:51,448 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:51,448 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:51,448 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=134.18 KB heapSize=374.77 KB 2023-04-05 18:53:51,449 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:51,449 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:51,449 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:51,449 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:51,449 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:51,449 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:51,455 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/B of be0aa03376eb89131764dd34306ebfb5 into 2ea9de25f9474eb694736ea7297adf45(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:51,455 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:51,455 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/B, priority=13, startTime=1680720831348; duration=0sec 2023-04-05 18:53:51,456 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:51,456 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:51,457 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:51,458 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/C is initiating minor compaction (all files) 2023-04-05 18:53:51,458 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/C in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:51,458 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/b3dbd554bf77452fbbacdbeb5d7706d2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ba937be173045d1a83c93fc11ce7f52, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a46c44e642a5447a96f87d9c53761d09] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=36.2 K 2023-04-05 18:53:51,458 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting b3dbd554bf77452fbbacdbeb5d7706d2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680720829176 2023-04-05 18:53:51,459 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 7ba937be173045d1a83c93fc11ce7f52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680720830419 2023-04-05 18:53:51,460 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a46c44e642a5447a96f87d9c53761d09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680720830533 2023-04-05 18:53:51,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:51,486 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:51,487 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#C#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:51,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720891501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720891503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,512 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/638f0c0e9beb42d9aa09aba7416e474e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/638f0c0e9beb42d9aa09aba7416e474e 2023-04-05 18:53:51,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42946 deadline: 1680720891513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720891514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720891514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,519 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/C of be0aa03376eb89131764dd34306ebfb5 into 638f0c0e9beb42d9aa09aba7416e474e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:51,520 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:51,520 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/C, priority=13, startTime=1680720831348; duration=0sec 2023-04-05 18:53:51,520 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:51,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720891610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720891611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:51,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720891817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720891816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:51,829 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/91b6a9c1532b4f62868a93c35df88477 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/91b6a9c1532b4f62868a93c35df88477 2023-04-05 18:53:51,839 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/A of be0aa03376eb89131764dd34306ebfb5 into 91b6a9c1532b4f62868a93c35df88477(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:51,839 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:51,840 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/A, priority=13, startTime=1680720831348; duration=0sec 2023-04-05 18:53:51,840 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:51,911 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/6dc3df3f272e4228a0d9a63db5b57b25 2023-04-05 18:53:51,936 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/50a568afa43c4fb3a490c5d21aa1ba4b 2023-04-05 18:53:51,962 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/cd9178152a4b49e89d9e3ff6e8592fee 2023-04-05 18:53:51,993 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/6dc3df3f272e4228a0d9a63db5b57b25 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6dc3df3f272e4228a0d9a63db5b57b25 2023-04-05 18:53:52,005 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6dc3df3f272e4228a0d9a63db5b57b25, entries=150, sequenceid=524, filesize=11.9 K 2023-04-05 18:53:52,006 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/50a568afa43c4fb3a490c5d21aa1ba4b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/50a568afa43c4fb3a490c5d21aa1ba4b 2023-04-05 18:53:52,014 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/50a568afa43c4fb3a490c5d21aa1ba4b, entries=150, sequenceid=524, filesize=11.9 K 2023-04-05 18:53:52,015 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/cd9178152a4b49e89d9e3ff6e8592fee as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/cd9178152a4b49e89d9e3ff6e8592fee 2023-04-05 18:53:52,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720892022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720892024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42946 deadline: 1680720892026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,032 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/cd9178152a4b49e89d9e3ff6e8592fee, entries=150, sequenceid=524, filesize=11.9 K 2023-04-05 18:53:52,033 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~374.72 KB/383712, currentSize=46.96 KB/48090 for be0aa03376eb89131764dd34306ebfb5 in 585ms, sequenceid=524, compaction requested=false 2023-04-05 18:53:52,036 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:52,036 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:52,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,079 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:52,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:52,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bd48293[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:52,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:52,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:52,123 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:53:52,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:52,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:52,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:52,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:52,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:52,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:52,133 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,133 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:52,133 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:52,133 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:52,172 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:52,172 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:52,172 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,172 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:52,173 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:52,173 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:52,173 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:52,173 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:52,173 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:52,173 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:52,173 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:52,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720892171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720892176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,225 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,225 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:52,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,225 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:52,226 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,229 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720892279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720892279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,294 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,294 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:52,294 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:52,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:52,294 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:52,294 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:52,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:52,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:52,294 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:52,295 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,295 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,295 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,295 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:52,295 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:52,295 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:52,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:52,395 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:52,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:52,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:52,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:52,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:52,400 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:52,400 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:52,400 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:52,400 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:52,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:52,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:52,472 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:52,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:52,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:52,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:52,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:52,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720892482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:52,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:52,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720892483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:52,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:52,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:52,492 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:52,493 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:52,495 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:52,495 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:52,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:52,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:52,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:52,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:52,533 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,533 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:52,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:52,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:52,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75999f68[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:52,536 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:52,536 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,558 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/6d8e9fb06bcf41b9997c013f63ac528e 2023-04-05 18:53:52,579 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:52,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:52,648 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:52,648 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:52,653 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:52,653 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:52,653 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:52,653 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:52,653 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:52,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:52,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:52,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:52,705 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,740 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,740 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:52,741 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:52,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:52,740 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:52,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:52,741 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:52,741 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,741 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,741 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,741 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:52,741 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:52,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:52,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:52,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:52,741 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:52,741 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:52,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720892791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:52,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720892791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,842 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:52,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:52,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:52,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:52,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:52,845 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:52,845 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:52,845 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:52,845 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:52,846 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,846 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:52,886 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:52,886 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:52,887 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,887 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:52,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:52,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:52,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:52,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:52,891 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:52,891 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:52,891 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:52,891 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:52,891 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:52,891 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:52,891 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:52,891 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:52,891 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,891 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:52,891 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:52,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:52,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:52,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:52,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae977a7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:52,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:52,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/ccdaaf2804c8488aa32fe4f554866b8e 2023-04-05 18:53:52,994 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:52,995 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:52,995 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:53,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:53,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720893028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:53,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42946 deadline: 1680720893035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:53,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720893035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,225 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,225 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,225 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,225 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:53,228 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:53,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:53,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:53,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:53,229 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:53,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,246 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,263 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,264 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:53,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:53,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:53,264 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:53,264 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:53,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,265 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,265 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720893296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720893296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,365 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:53,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:53,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:53,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:53,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:53,368 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:53,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:53,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:53,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:53,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:53,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:53,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:53,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:53,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:53,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:53,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@465611d6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:53,376 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:53,376 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,377 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:53,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:53,377 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:53,377 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:53,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:53,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:53,383 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:53,383 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:53,383 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:53,383 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:53,383 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:53,384 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,384 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,384 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,395 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,403 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/bd52f11a2b2f4b16a773501319f57b90 2023-04-05 18:53:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,410 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,413 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:53,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:53,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,414 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:53,414 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:53,417 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/6d8e9fb06bcf41b9997c013f63ac528e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6d8e9fb06bcf41b9997c013f63ac528e 2023-04-05 18:53:53,423 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6d8e9fb06bcf41b9997c013f63ac528e, entries=200, sequenceid=539, filesize=14.3 K 2023-04-05 18:53:53,425 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/ccdaaf2804c8488aa32fe4f554866b8e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/ccdaaf2804c8488aa32fe4f554866b8e 2023-04-05 18:53:53,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/ccdaaf2804c8488aa32fe4f554866b8e, entries=150, sequenceid=539, filesize=11.9 K 2023-04-05 18:53:53,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/bd52f11a2b2f4b16a773501319f57b90 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bd52f11a2b2f4b16a773501319f57b90 2023-04-05 18:53:53,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bd52f11a2b2f4b16a773501319f57b90, entries=150, sequenceid=539, filesize=11.9 K 2023-04-05 18:53:53,447 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for be0aa03376eb89131764dd34306ebfb5 in 1324ms, sequenceid=539, compaction requested=true 2023-04-05 18:53:53,447 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:53,447 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:53,447 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:53,447 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:53,447 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:53,448 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:53,449 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:53,449 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/B is initiating minor compaction (all files) 2023-04-05 18:53:53,449 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:53,449 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/B in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,449 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/A is initiating minor compaction (all files) 2023-04-05 18:53:53,449 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2ea9de25f9474eb694736ea7297adf45, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/50a568afa43c4fb3a490c5d21aa1ba4b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/ccdaaf2804c8488aa32fe4f554866b8e] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=36.3 K 2023-04-05 18:53:53,449 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/A in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,449 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/91b6a9c1532b4f62868a93c35df88477, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6dc3df3f272e4228a0d9a63db5b57b25, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6d8e9fb06bcf41b9997c013f63ac528e] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=38.7 K 2023-04-05 18:53:53,450 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 2ea9de25f9474eb694736ea7297adf45, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680720830533 2023-04-05 18:53:53,451 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 91b6a9c1532b4f62868a93c35df88477, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680720830533 2023-04-05 18:53:53,451 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 50a568afa43c4fb3a490c5d21aa1ba4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1680720830878 2023-04-05 18:53:53,451 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6dc3df3f272e4228a0d9a63db5b57b25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1680720830878 2023-04-05 18:53:53,452 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ccdaaf2804c8488aa32fe4f554866b8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680720831501 2023-04-05 18:53:53,452 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6d8e9fb06bcf41b9997c013f63ac528e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680720831499 2023-04-05 18:53:53,464 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#B#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:53,465 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#A#compaction#64 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:53,514 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:53,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:53,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:53,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:53,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:53,517 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:53,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:53,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:53,518 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:53,518 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,518 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,519 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,520 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,520 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:53,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:53,521 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:53,521 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:53,521 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:53,521 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:53,521 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:53,521 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,521 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:53,522 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:53:53,522 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:53,522 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:53,522 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:53,522 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:53,522 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:53,522 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:53,530 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/16d6965726c3490c8b56135f2f0e0203 2023-04-05 18:53:53,541 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/f6b1bd92f5a84222ab8ce1a1c4a06b47 2023-04-05 18:53:53,555 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/137d75df309f4825a9bdd9c494d539af 2023-04-05 18:53:53,561 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/16d6965726c3490c8b56135f2f0e0203 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/16d6965726c3490c8b56135f2f0e0203 2023-04-05 18:53:53,566 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/16d6965726c3490c8b56135f2f0e0203, entries=150, sequenceid=559, filesize=11.9 K 2023-04-05 18:53:53,567 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/f6b1bd92f5a84222ab8ce1a1c4a06b47 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/f6b1bd92f5a84222ab8ce1a1c4a06b47 2023-04-05 18:53:53,574 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/f6b1bd92f5a84222ab8ce1a1c4a06b47, entries=150, sequenceid=559, filesize=11.9 K 2023-04-05 18:53:53,575 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/137d75df309f4825a9bdd9c494d539af as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/137d75df309f4825a9bdd9c494d539af 2023-04-05 18:53:53,579 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/137d75df309f4825a9bdd9c494d539af, entries=150, sequenceid=559, filesize=11.9 K 2023-04-05 18:53:53,581 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for be0aa03376eb89131764dd34306ebfb5 in 59ms, sequenceid=559, compaction requested=true 2023-04-05 18:53:53,581 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:53,581 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:53,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:53,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:53,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:53,583 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:53,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2979c395[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:53,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:53,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,586 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:53,587 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,587 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,589 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:53,592 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:53,592 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:53,592 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:53,592 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:53,592 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:53,595 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,596 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,597 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:53,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:53,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,600 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:53,600 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,600 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:53,600 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,600 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,600 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,700 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:53,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:53,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:53,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:53,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:53,704 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:53,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:53,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:53,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:53,705 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,705 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:53,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:53,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:53,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:53,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:53,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:53,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:53,709 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,709 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:53,710 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:53,710 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:53,713 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,713 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:53,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a26b0d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:53,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:53,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:53,717 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,717 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:53,718 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,719 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:53,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:53,721 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:53,721 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:53,721 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:53,721 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:53,721 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:53,725 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,725 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,725 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,726 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,727 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,734 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,734 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,734 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,735 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,735 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,735 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,735 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:53,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:53,735 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,735 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,735 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,735 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:53,735 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:53,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,835 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:53,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:53,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:53,839 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:53,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:53,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:53,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:53,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,841 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,841 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,841 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,841 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:53,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:53,842 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:53,843 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,843 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:53,843 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:53,843 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:53,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,845 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,846 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,846 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@117772f1[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:53,847 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:53,847 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,849 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:53,849 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:53,849 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:53,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:53,850 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,850 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,850 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,850 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:53,852 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:53,852 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:53,852 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:53,852 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:53,852 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:53,854 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,854 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,854 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,855 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,856 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,859 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:53,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:53,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,860 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:53,860 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:53,860 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,860 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,884 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/7bf139a3074b4a1eb8ab4f2ce42d2fe4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/7bf139a3074b4a1eb8ab4f2ce42d2fe4 2023-04-05 18:53:53,888 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/eb282a6a691b4efcbb627f57cea32829 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/eb282a6a691b4efcbb627f57cea32829 2023-04-05 18:53:53,891 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/A of be0aa03376eb89131764dd34306ebfb5 into 7bf139a3074b4a1eb8ab4f2ce42d2fe4(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:53,891 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:53,891 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/A, priority=13, startTime=1680720833447; duration=0sec 2023-04-05 18:53:53,891 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:53,891 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:53,893 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:53,893 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/C is initiating minor compaction (all files) 2023-04-05 18:53:53,893 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/C in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,893 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/638f0c0e9beb42d9aa09aba7416e474e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/cd9178152a4b49e89d9e3ff6e8592fee, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bd52f11a2b2f4b16a773501319f57b90, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/137d75df309f4825a9bdd9c494d539af] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=48.2 K 2023-04-05 18:53:53,894 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 638f0c0e9beb42d9aa09aba7416e474e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680720830533 2023-04-05 18:53:53,894 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting cd9178152a4b49e89d9e3ff6e8592fee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1680720830878 2023-04-05 18:53:53,894 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/B of be0aa03376eb89131764dd34306ebfb5 into eb282a6a691b4efcbb627f57cea32829(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:53,894 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:53,894 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting bd52f11a2b2f4b16a773501319f57b90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680720831501 2023-04-05 18:53:53,894 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/B, priority=13, startTime=1680720833447; duration=0sec 2023-04-05 18:53:53,895 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:53,895 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 137d75df309f4825a9bdd9c494d539af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1680720832159 2023-04-05 18:53:53,905 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#C#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:53,928 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/631c71bade9c4ebd93ec88bccd8bbac6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/631c71bade9c4ebd93ec88bccd8bbac6 2023-04-05 18:53:53,935 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/C of be0aa03376eb89131764dd34306ebfb5 into 631c71bade9c4ebd93ec88bccd8bbac6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:53,935 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:53,935 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/C, priority=12, startTime=1680720833448; duration=0sec 2023-04-05 18:53:53,935 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:53,960 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:53,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:53,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:53,964 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:53,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:53,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:53,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:53,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:53,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:53,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:53,968 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,968 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:53,968 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:53,968 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:53,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:53,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,970 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:53,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675919a8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:53,972 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:53,972 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:53,974 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,974 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:53,977 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:53,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:53,980 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:53,980 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:53,980 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:53,980 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:53,980 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:53,982 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,982 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:53,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,982 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:53,983 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:53,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:53,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:53,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:53,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:53,986 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:53,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,987 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:53,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:53,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:53,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:53,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:53,987 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:53,987 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:53,987 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:54,087 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:54,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:54,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:54,092 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:54,092 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,092 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:54,092 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:54,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,096 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,096 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:54,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:54,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:54,097 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:54,097 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,097 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:54,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:54,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:54,098 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,098 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:54,098 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:54,098 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:54,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:54,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:54,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:54,100 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,100 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,100 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,100 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:54,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1bbd[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:54,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:54,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,104 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,104 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:54,104 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:54,104 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:54,105 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,105 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:54,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,106 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,106 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:54,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:54,109 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:54,109 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:54,109 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:54,109 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:54,109 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:54,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,112 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,116 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,121 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,121 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,121 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,122 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,122 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:54,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:54,122 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,122 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,122 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:54,122 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:54,122 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,122 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,222 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:54,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:54,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:54,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:54,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:54,227 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:54,227 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,227 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:54,227 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:54,227 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,227 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,230 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,230 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,234 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:54,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:54,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:54,236 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:54,236 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,236 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:54,237 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,237 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:54,237 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:54,237 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:54,239 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,239 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:54,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a2e2b8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:54,243 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:54,243 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:54,245 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,245 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:54,246 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,246 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:54,246 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,246 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:54,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:54,249 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:54,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:54,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:54,250 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:54,250 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:54,254 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,254 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,254 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,255 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,259 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:54,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:54,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,259 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:54,260 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:54,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:54,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:54,323 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:53:54,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:54,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:54,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:54,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:54,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:54,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:54,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:54,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720894353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680720834333, max timestamp=1680720834353Num uniques 859; 2023-04-05 18:53:54,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-05 18:53:54,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680720834333, max timestamp=1680720834353Num uniques 859; 2023-04-05 18:53:54,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-05 18:53:54,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720834346, max timestamp=1680720834353Num uniques 859; 2023-04-05 18:53:54,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:53:54,360 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:54,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:54,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:54,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:54,364 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:54,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:54,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:54,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,367 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:54,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:54,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:54,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:54,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:54,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,375 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,375 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2968f8c7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:54,376 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:54,376 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,378 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:54,378 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:54,378 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:54,378 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,378 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:54,379 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:54,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:54,382 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:54,382 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:54,382 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:54,383 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:54,383 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:54,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,386 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,388 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,391 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,391 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,391 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:54,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:54,391 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,391 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:54,392 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:54,392 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:54,392 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,392 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,392 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,392 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,392 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:54,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720894424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:54,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720894456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,492 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:54,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:54,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:54,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:54,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:54,499 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:54,499 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,499 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:54,499 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:54,499 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,499 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,501 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,501 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,501 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,501 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:54,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:54,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:54,503 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:54,503 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,503 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:54,504 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,504 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:54,504 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:54,504 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:54,507 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,507 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,507 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,507 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:54,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@653301b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:54,509 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:54,509 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:54,510 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,511 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:54,512 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:54,512 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,512 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,512 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:54,516 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:54,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:54,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:54,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:54,516 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:54,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,520 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,523 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:54,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720894529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,532 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,532 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,532 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:54,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:54,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,532 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,532 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,532 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,533 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,533 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,533 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,533 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,533 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:54,534 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:54,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:54,634 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:54,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:54,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:54,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:54,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:54,637 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:54,637 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,637 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:54,638 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:54,638 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,638 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,642 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,642 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,642 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,642 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:54,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:54,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:54,643 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:54,643 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,643 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:54,644 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,644 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:54,644 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:54,644 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:54,644 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:54,647 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,647 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,647 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,647 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:54,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0d559f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:54,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:54,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,650 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:54,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:54,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:54,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:54,654 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:54,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:54,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:54,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:54,655 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,658 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,662 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,662 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,662 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,662 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,662 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,662 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:54,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:54,662 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,663 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,663 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,663 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,663 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:54,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:54,663 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:54,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720894662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:54,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720894732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/bc659b99038d449bb88fa192841fee26 2023-04-05 18:53:54,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/29c0c196eeae4ef9954b5ea6191a322d 2023-04-05 18:53:54,763 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:54,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:54,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:54,767 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:54,767 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,767 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:54,767 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:54,768 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,768 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,769 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,770 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,770 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:54,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:54,771 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,771 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:54,772 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:54,772 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:54,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,773 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,774 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,774 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aaecbc4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:54,775 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:54,775 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,778 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,778 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:54,778 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:54,778 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:54,778 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,778 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:54,779 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:54,779 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,779 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,780 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:54,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:54,782 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:54,782 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:54,782 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:54,782 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:54,782 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:54,784 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,785 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,788 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:54,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:54,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,789 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,789 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:54,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,789 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:54,789 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:54,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,889 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:54,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:54,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:54,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:54,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:54,893 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:54,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:54,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:54,894 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,894 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,896 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,896 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,897 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,897 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:54,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:54,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:54,898 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,898 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:54,898 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:54,898 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:54,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:54,900 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,902 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,902 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:54,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:54,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@246eac96[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:54,902 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:54,902 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,903 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:54,903 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:54,903 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:54,904 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,906 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:54,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:54,909 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:54,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:54,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:54,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:54,909 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:54,911 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,911 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:54,911 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:54,912 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:54,913 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:54,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:54,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:54,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:54,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:54,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:54,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:54,916 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:54,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:54,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:54,916 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:54,916 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:54,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:54,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:54,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720894965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,016 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:55,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:55,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:55,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:55,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:55,020 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:55,020 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:55,020 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:55,020 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:55,020 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,021 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,022 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:55,022 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:55,022 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,022 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:55,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:55,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:55,023 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:55,023 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:55,023 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:55,024 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:55,024 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:55,024 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:55,024 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:55,024 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:55,024 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:55,024 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:55,024 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:55,024 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,024 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:55,024 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:55,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,025 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,026 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,026 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:55,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ba293e8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:55,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:55,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,032 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:55,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:55,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:55,032 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,032 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:55,033 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:55,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,034 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:55,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:55,036 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:55,036 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:55,036 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:55,036 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:55,036 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:55,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:55,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720895035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:55,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720895036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,039 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,039 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:53:55,039 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,039 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:55,040 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,046 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:55,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:55,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:55,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:55,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:55,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:55,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,047 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,047 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:55,047 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:55,047 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:55,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:55,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:55,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720895049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,054 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:53:55,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:55,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42946 deadline: 1680720895051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,056 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:53:55,147 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:55,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:55,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:55,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:55,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:55,151 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:55,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:55,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:55,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:55,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,152 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:55,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:55,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:55,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:55,155 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:55,155 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:55,155 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:55,155 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:55,157 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,157 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,157 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,157 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:55,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f3798ae[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:55,159 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:55,159 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,160 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,160 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:55,160 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:55,160 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:55,161 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,161 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:55,162 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:55,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:55,165 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:55,166 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:55,166 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:55,166 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:55,166 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:55,167 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/ba9937ebe57d4834b10d7597587efcd0 2023-04-05 18:53:55,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/bc659b99038d449bb88fa192841fee26 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/bc659b99038d449bb88fa192841fee26 2023-04-05 18:53:55,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:55,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,178 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/bc659b99038d449bb88fa192841fee26, entries=250, sequenceid=578, filesize=16.7 K 2023-04-05 18:53:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/29c0c196eeae4ef9954b5ea6191a322d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/29c0c196eeae4ef9954b5ea6191a322d 2023-04-05 18:53:55,181 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:55,182 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:55,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:55,182 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:55,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:55,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:55,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:55,182 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,182 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,182 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,182 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,182 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,182 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:55,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:55,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/29c0c196eeae4ef9954b5ea6191a322d, entries=150, sequenceid=578, filesize=11.9 K 2023-04-05 18:53:55,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/ba9937ebe57d4834b10d7597587efcd0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/ba9937ebe57d4834b10d7597587efcd0 2023-04-05 18:53:55,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:55,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/ba9937ebe57d4834b10d7597587efcd0, entries=150, sequenceid=578, filesize=11.9 K 2023-04-05 18:53:55,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.94 KB/135110 for be0aa03376eb89131764dd34306ebfb5 in 873ms, sequenceid=578, compaction requested=true 2023-04-05 18:53:55,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:55,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:55,196 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:55,196 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:55,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:55,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:55,198 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:55,198 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/B is initiating minor compaction (all files) 2023-04-05 18:53:55,198 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42154 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:55,198 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/B in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:55,198 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/A is initiating minor compaction (all files) 2023-04-05 18:53:55,198 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/eb282a6a691b4efcbb627f57cea32829, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/f6b1bd92f5a84222ab8ce1a1c4a06b47, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/29c0c196eeae4ef9954b5ea6191a322d] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=36.4 K 2023-04-05 18:53:55,198 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/A in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:55,198 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/7bf139a3074b4a1eb8ab4f2ce42d2fe4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/16d6965726c3490c8b56135f2f0e0203, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/bc659b99038d449bb88fa192841fee26] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=41.2 K 2023-04-05 18:53:55,199 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting eb282a6a691b4efcbb627f57cea32829, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680720831501 2023-04-05 18:53:55,199 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7bf139a3074b4a1eb8ab4f2ce42d2fe4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680720831501 2023-04-05 18:53:55,199 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting f6b1bd92f5a84222ab8ce1a1c4a06b47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1680720832159 2023-04-05 18:53:55,199 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 16d6965726c3490c8b56135f2f0e0203, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1680720832159 2023-04-05 18:53:55,200 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 29c0c196eeae4ef9954b5ea6191a322d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680720834319 2023-04-05 18:53:55,200 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting bc659b99038d449bb88fa192841fee26, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680720834313 2023-04-05 18:53:55,209 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#A#compaction#72 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:55,209 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#B#compaction#73 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:55,282 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:55,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:55,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:55,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:55,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:55,286 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:55,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:55,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:55,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:55,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,288 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:55,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:55,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:55,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:55,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:55,291 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:55,291 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:55,291 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=131.94 KB heapSize=365.84 KB 2023-04-05 18:53:55,291 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:55,291 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:55,291 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:55,291 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:55,291 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:55,292 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:55,300 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/f3dc423e3a464737b9a545907a0ba959 2023-04-05 18:53:55,314 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/5a2fb3c40f50492ab3134f3c9234a808 2023-04-05 18:53:55,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:55,473 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:55,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:55,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720895492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:55,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720895540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:55,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720895596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,626 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/f0bcf8d69320402e8ec3e76ea6e1314e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f0bcf8d69320402e8ec3e76ea6e1314e 2023-04-05 18:53:55,631 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/27975b32d5a848e6bf11b695e5ad9dd6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/27975b32d5a848e6bf11b695e5ad9dd6 2023-04-05 18:53:55,635 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/A of be0aa03376eb89131764dd34306ebfb5 into f0bcf8d69320402e8ec3e76ea6e1314e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:55,635 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:55,635 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/A, priority=13, startTime=1680720835196; duration=0sec 2023-04-05 18:53:55,635 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:55,635 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-05 18:53:55,637 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-05 18:53:55,637 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-05 18:53:55,637 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. because compaction request was cancelled 2023-04-05 18:53:55,653 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/B of be0aa03376eb89131764dd34306ebfb5 into 27975b32d5a848e6bf11b695e5ad9dd6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:55,653 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:55,653 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/B, priority=13, startTime=1680720835196; duration=0sec 2023-04-05 18:53:55,653 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:55,734 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/73a7f4e4ea8943b48b5abdb145b3afc3 2023-04-05 18:53:55,740 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/f3dc423e3a464737b9a545907a0ba959 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f3dc423e3a464737b9a545907a0ba959 2023-04-05 18:53:55,745 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f3dc423e3a464737b9a545907a0ba959, entries=150, sequenceid=612, filesize=11.9 K 2023-04-05 18:53:55,746 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/5a2fb3c40f50492ab3134f3c9234a808 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5a2fb3c40f50492ab3134f3c9234a808 2023-04-05 18:53:55,753 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5a2fb3c40f50492ab3134f3c9234a808, entries=150, sequenceid=612, filesize=11.9 K 2023-04-05 18:53:55,754 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/73a7f4e4ea8943b48b5abdb145b3afc3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/73a7f4e4ea8943b48b5abdb145b3afc3 2023-04-05 18:53:55,760 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/73a7f4e4ea8943b48b5abdb145b3afc3, entries=150, sequenceid=612, filesize=11.9 K 2023-04-05 18:53:55,761 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~365.80 KB/374576, currentSize=53.67 KB/54960 for be0aa03376eb89131764dd34306ebfb5 in 470ms, sequenceid=612, compaction requested=true 2023-04-05 18:53:55,761 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:55,761 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:55,761 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:55,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:55,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:55,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,764 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33fb8267[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:55,766 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:55,766 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,769 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:55,769 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:55,769 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:55,771 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:55,771 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:55,772 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:55,772 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:55,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:55,773 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:55,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:55,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:55,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:55,773 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:55,775 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,775 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,775 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,778 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,783 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:55,783 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:55,783 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:55,783 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:55,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:55,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:55,783 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:55,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:55,784 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,784 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,784 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,784 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,784 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,785 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:55,785 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:55,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:55,801 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:53:55,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:55,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:55,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:55,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:55,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:55,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:55,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:55,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720895880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,885 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:55,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:55,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:55,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:55,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:55,889 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:55,889 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:55,889 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:55,889 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:55,890 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,890 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,891 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:55,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:55,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,892 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,892 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:55,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:55,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:55,897 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:55,897 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:55,897 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:55,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:55,900 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:55,900 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:55,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:55,900 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:55,900 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:55,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:55,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:55,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:55,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,904 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,904 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,904 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:55,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a09292[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:55,906 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:55,906 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,907 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:55,907 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:55,908 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,908 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:55,911 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:55,912 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:55,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:55,915 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:55,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:55,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:55,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:55,915 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:55,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,916 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:55,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,917 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:55,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,918 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,920 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:55,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:55,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,923 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:55,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:55,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:55,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:55,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:55,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:55,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:55,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:55,923 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:55,923 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:55,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:55,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:55,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720895984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,024 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:56,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:56,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:56,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:56,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:56,027 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:56,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:56,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:56,028 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,028 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,030 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,031 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,031 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:56,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:56,033 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,033 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:56,033 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:56,033 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:56,035 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,035 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,035 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,035 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4db270fd[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:56,037 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:56,037 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,038 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:56,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:56,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:56,038 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,038 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:56,039 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:56,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:56,042 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:56,042 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:56,042 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:56,042 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:56,042 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:56,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,046 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,049 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,049 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,049 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:56,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:56,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:56,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,050 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:56,050 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:56,050 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:56,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:56,150 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:56,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:56,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:56,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:56,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:56,156 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:56,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:56,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:56,157 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,157 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,158 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,158 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,158 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,159 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:56,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:56,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:56,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:56,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:56,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:56,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:56,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:56,166 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,166 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:56,166 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:56,166 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:56,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:56,167 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,167 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:56,167 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:56,169 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,169 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,169 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,169 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:56,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef3630d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:56,171 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:56,171 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:56,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:56,176 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:56,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:56,180 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:56,180 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:56,180 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:56,180 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:56,180 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:56,183 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,183 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:56,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,185 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,186 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:56,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:56,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:56,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,190 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,190 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,190 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,190 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,190 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,190 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:56,190 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:56,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:56,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720896189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:56,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/cc7a56d6a2ff4034ac553504965602fa 2023-04-05 18:53:56,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/5ff036f18b234e77b8ad94dc2a04c6ac 2023-04-05 18:53:56,290 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:56,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:56,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:56,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:56,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:56,295 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:56,295 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,295 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:56,296 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:56,296 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,296 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,297 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,298 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,298 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:56,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:56,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:56,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:56,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:56,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:56,304 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:56,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f42e0e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:56,307 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:56,307 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:56,308 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,308 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:56,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:56,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:56,313 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:56,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:56,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:56,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:56,313 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:56,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:56,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:56,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,317 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:56,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:56,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:56,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:56,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:56,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,324 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,324 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,324 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,324 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:56,324 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:56,324 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:56,424 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:56,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:56,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:56,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:56,427 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:56,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:56,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:56,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,429 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,430 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,430 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:56,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:56,431 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:56,431 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,431 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:56,432 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,432 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:56,432 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:56,432 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:56,433 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:56,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a61aa0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:56,435 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:56,435 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:56,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:56,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:56,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:56,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,439 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,439 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,439 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:56,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:56,441 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:56,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:56,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:56,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:56,441 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:56,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:56,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,447 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:56,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:56,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:56,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,448 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:56,448 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:56,448 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:56,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:56,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:56,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720896496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,548 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:56,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:56,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:56,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720896547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:56,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:56,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:56,551 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:56,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:56,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:56,552 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,552 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,559 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,559 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,560 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:56,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:56,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:56,561 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:56,561 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,561 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:56,561 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:56,561 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:56,561 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,561 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:56,562 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:56,562 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,562 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:56,562 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:56,562 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,562 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:56,562 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:56,563 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:56,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@294db931[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:56,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:56,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,566 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:56,566 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,566 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:56,567 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:56,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:56,570 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:56,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:56,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:56,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:56,570 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:56,572 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,572 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:56,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,572 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:56,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:56,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,574 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,576 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,576 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,577 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:56,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:56,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:56,576 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,576 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:56,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,577 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:56,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:56,577 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:56,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:56,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/171453c3ad98449ca1a45e9edaf81e28 2023-04-05 18:53:56,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/cc7a56d6a2ff4034ac553504965602fa as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/cc7a56d6a2ff4034ac553504965602fa 2023-04-05 18:53:56,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/cc7a56d6a2ff4034ac553504965602fa, entries=200, sequenceid=626, filesize=14.3 K 2023-04-05 18:53:56,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/5ff036f18b234e77b8ad94dc2a04c6ac as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5ff036f18b234e77b8ad94dc2a04c6ac 2023-04-05 18:53:56,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5ff036f18b234e77b8ad94dc2a04c6ac, entries=150, sequenceid=626, filesize=11.9 K 2023-04-05 18:53:56,663 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/171453c3ad98449ca1a45e9edaf81e28 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/171453c3ad98449ca1a45e9edaf81e28 2023-04-05 18:53:56,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/171453c3ad98449ca1a45e9edaf81e28, entries=150, sequenceid=626, filesize=11.9 K 2023-04-05 18:53:56,668 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for be0aa03376eb89131764dd34306ebfb5 in 867ms, sequenceid=626, compaction requested=true 2023-04-05 18:53:56,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:56,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:56,669 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:56,669 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:53:56,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:56,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:56,670 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39816 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:56,670 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/A is initiating minor compaction (all files) 2023-04-05 18:53:56,670 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/A in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,670 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f0bcf8d69320402e8ec3e76ea6e1314e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f3dc423e3a464737b9a545907a0ba959, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/cc7a56d6a2ff4034ac553504965602fa] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=38.9 K 2023-04-05 18:53:56,670 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:53:56,671 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/B is initiating minor compaction (all files) 2023-04-05 18:53:56,671 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/B in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,671 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/27975b32d5a848e6bf11b695e5ad9dd6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5a2fb3c40f50492ab3134f3c9234a808, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5ff036f18b234e77b8ad94dc2a04c6ac] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=36.5 K 2023-04-05 18:53:56,671 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting f0bcf8d69320402e8ec3e76ea6e1314e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680720834319 2023-04-05 18:53:56,671 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 27975b32d5a848e6bf11b695e5ad9dd6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680720834319 2023-04-05 18:53:56,671 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting f3dc423e3a464737b9a545907a0ba959, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1680720834380 2023-04-05 18:53:56,672 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting cc7a56d6a2ff4034ac553504965602fa, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1680720835475 2023-04-05 18:53:56,672 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 5a2fb3c40f50492ab3134f3c9234a808, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1680720834380 2023-04-05 18:53:56,673 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 5ff036f18b234e77b8ad94dc2a04c6ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1680720835479 2023-04-05 18:53:56,677 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:56,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:56,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:56,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:56,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:56,683 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:56,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:56,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:56,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,684 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,684 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#A#compaction#80 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:56,685 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:56,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:56,685 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#B#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:56,686 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,686 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:56,686 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:56,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:56,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:56,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:56,687 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:56,687 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:56,687 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:56,687 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:56,687 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:56,687 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,687 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:56,687 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:53:56,688 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:56,688 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:56,688 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:56,688 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:56,688 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:56,688 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:56,705 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/a381afb03ef64e6faec9a58e38cf41a1 2023-04-05 18:53:56,706 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/ff2e525bf60e46efbd1d60856502d2cc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ff2e525bf60e46efbd1d60856502d2cc 2023-04-05 18:53:56,713 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/09f5c85718044a10a2da7dcb6d6c7402 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/09f5c85718044a10a2da7dcb6d6c7402 2023-04-05 18:53:56,714 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/A of be0aa03376eb89131764dd34306ebfb5 into ff2e525bf60e46efbd1d60856502d2cc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:56,715 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:56,715 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/A, priority=13, startTime=1680720836668; duration=0sec 2023-04-05 18:53:56,715 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:56,715 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:53:56,717 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:53:56,717 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/C is initiating minor compaction (all files) 2023-04-05 18:53:56,718 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/C in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:56,718 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/631c71bade9c4ebd93ec88bccd8bbac6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/ba9937ebe57d4834b10d7597587efcd0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/73a7f4e4ea8943b48b5abdb145b3afc3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/171453c3ad98449ca1a45e9edaf81e28] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=48.3 K 2023-04-05 18:53:56,718 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 631c71bade9c4ebd93ec88bccd8bbac6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1680720832159 2023-04-05 18:53:56,719 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting ba9937ebe57d4834b10d7597587efcd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680720834319 2023-04-05 18:53:56,719 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/B of be0aa03376eb89131764dd34306ebfb5 into 09f5c85718044a10a2da7dcb6d6c7402(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:56,719 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 73a7f4e4ea8943b48b5abdb145b3afc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1680720834380 2023-04-05 18:53:56,720 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:56,720 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/B, priority=13, startTime=1680720836669; duration=0sec 2023-04-05 18:53:56,720 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 171453c3ad98449ca1a45e9edaf81e28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1680720835479 2023-04-05 18:53:56,720 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:56,722 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/6efddd297f0540c2ac35f752daff5279 2023-04-05 18:53:56,748 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#C#compaction#85 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:56,778 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/88d7209620d94220b22f6b03c96d14d3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/88d7209620d94220b22f6b03c96d14d3 2023-04-05 18:53:56,784 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/C of be0aa03376eb89131764dd34306ebfb5 into 88d7209620d94220b22f6b03c96d14d3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:56,784 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:56,784 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/C, priority=12, startTime=1680720836669; duration=0sec 2023-04-05 18:53:56,784 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:57,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:57,001 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:57,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:57,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720897019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:57,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720897121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,161 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/6a8ef3adc1a44853892700c5cec8333b 2023-04-05 18:53:57,167 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/a381afb03ef64e6faec9a58e38cf41a1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/a381afb03ef64e6faec9a58e38cf41a1 2023-04-05 18:53:57,178 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/a381afb03ef64e6faec9a58e38cf41a1, entries=150, sequenceid=646, filesize=11.9 K 2023-04-05 18:53:57,179 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/6efddd297f0540c2ac35f752daff5279 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6efddd297f0540c2ac35f752daff5279 2023-04-05 18:53:57,185 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6efddd297f0540c2ac35f752daff5279, entries=150, sequenceid=646, filesize=11.9 K 2023-04-05 18:53:57,185 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/6a8ef3adc1a44853892700c5cec8333b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6a8ef3adc1a44853892700c5cec8333b 2023-04-05 18:53:57,190 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6a8ef3adc1a44853892700c5cec8333b, entries=150, sequenceid=646, filesize=11.9 K 2023-04-05 18:53:57,191 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for be0aa03376eb89131764dd34306ebfb5 in 504ms, sequenceid=646, compaction requested=false 2023-04-05 18:53:57,191 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:57,191 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:57,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cc34c1e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:57,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:57,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:57,196 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,196 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:57,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:57,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:57,198 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:57,200 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:57,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:57,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:57,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:57,201 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:57,204 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,204 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,204 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:57,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:57,204 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:57,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:57,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:57,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:57,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:57,210 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:57,211 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:57,211 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:57,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:57,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,311 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:57,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:57,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:57,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:57,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:57,314 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:57,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:57,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:57,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:57,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:57,316 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:57,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:57,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:57,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:57,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:57,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:57,317 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:57,318 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:57,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:57,324 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:57,333 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/37aa12cd6eae4eed96511de819fef2c8 2023-04-05 18:53:57,358 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/9f7504c6d68a4ea69ab9ec91f2477bf3 2023-04-05 18:53:57,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:57,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720897385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:57,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720897490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:57,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720897693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,774 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/1edc0dc85d7e4bb89930c789476dcda8 2023-04-05 18:53:57,782 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/37aa12cd6eae4eed96511de819fef2c8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/37aa12cd6eae4eed96511de819fef2c8 2023-04-05 18:53:57,789 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/37aa12cd6eae4eed96511de819fef2c8, entries=150, sequenceid=661, filesize=11.9 K 2023-04-05 18:53:57,790 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/9f7504c6d68a4ea69ab9ec91f2477bf3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/9f7504c6d68a4ea69ab9ec91f2477bf3 2023-04-05 18:53:57,798 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/9f7504c6d68a4ea69ab9ec91f2477bf3, entries=150, sequenceid=661, filesize=11.9 K 2023-04-05 18:53:57,799 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/1edc0dc85d7e4bb89930c789476dcda8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1edc0dc85d7e4bb89930c789476dcda8 2023-04-05 18:53:57,805 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1edc0dc85d7e4bb89930c789476dcda8, entries=150, sequenceid=661, filesize=11.9 K 2023-04-05 18:53:57,806 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for be0aa03376eb89131764dd34306ebfb5 in 488ms, sequenceid=661, compaction requested=true 2023-04-05 18:53:57,806 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:57,806 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,808 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:57,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf78fd6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:57,811 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,812 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,812 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,812 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:57,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:57,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:57,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:57,814 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:57,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:57,820 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:57,820 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:57,820 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:57,820 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:57,820 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:57,824 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,824 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:57,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,824 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:57,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,832 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,832 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:57,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:57,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:57,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:57,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:57,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:57,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:57,833 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:57,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,833 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:57,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:57,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:57,833 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:57,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:57,933 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:57,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:57,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:57,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:57,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:57,939 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:57,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:57,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:57,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:57,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:57,941 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:57,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:57,941 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:57,941 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:57,942 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:57,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:57,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:57,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:57,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:57,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:57,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:57,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:57,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:57,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:57,943 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:57,943 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:57,943 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:53:57,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:57,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:57,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:57,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:57,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:57,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:57,953 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/68d59a7c1a9347b5acf3db8c9f363efe 2023-04-05 18:53:57,978 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/756862c5b03e4dc1a304efdac9812089 2023-04-05 18:53:57,998 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/a11498c44338400488172dee365b3427 2023-04-05 18:53:58,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:58,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:58,006 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/68d59a7c1a9347b5acf3db8c9f363efe as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/68d59a7c1a9347b5acf3db8c9f363efe 2023-04-05 18:53:58,012 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/68d59a7c1a9347b5acf3db8c9f363efe, entries=150, sequenceid=681, filesize=11.9 K 2023-04-05 18:53:58,013 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/756862c5b03e4dc1a304efdac9812089 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/756862c5b03e4dc1a304efdac9812089 2023-04-05 18:53:58,025 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/756862c5b03e4dc1a304efdac9812089, entries=150, sequenceid=681, filesize=11.9 K 2023-04-05 18:53:58,026 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/a11498c44338400488172dee365b3427 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a11498c44338400488172dee365b3427 2023-04-05 18:53:58,032 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a11498c44338400488172dee365b3427, entries=150, sequenceid=681, filesize=11.9 K 2023-04-05 18:53:58,033 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=40.25 KB/41220 for be0aa03376eb89131764dd34306ebfb5 in 90ms, sequenceid=681, compaction requested=true 2023-04-05 18:53:58,033 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:58,033 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,035 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:58,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,036 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:53:58,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:58,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:58,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:58,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:58,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:58,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:58,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc3e9f2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:58,038 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:58,038 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,040 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:58,041 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:58,041 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:58,043 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:58,043 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,043 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:58,044 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:58,044 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:58,044 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:58,044 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:58,044 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:58,044 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:58,044 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:58,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,047 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,047 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,059 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/576a7f33fa0a4c02a75b1f92fdc59dd2 2023-04-05 18:53:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,062 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:58,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:58,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:58,062 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:58,062 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:58,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,079 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/2c1df05889044d84bee5951ee0a37eaa 2023-04-05 18:53:58,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720838066, max timestamp=1680720838092Num uniques 859; 2023-04-05 18:53:58,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720838066, max timestamp=1680720838092Num uniques 859; 2023-04-05 18:53:58,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:53:58,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:53:58,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720838066, max timestamp=1680720838092Num uniques 859; 2023-04-05 18:53:58,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:53:58,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:58,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720898145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,163 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:58,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:58,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:58,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:58,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:58,166 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:58,166 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:58,166 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:58,166 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:58,166 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,166 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,169 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,169 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,169 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,169 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:58,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:58,170 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,170 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:58,171 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:58,171 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:58,172 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f835b8a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:58,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:58,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,177 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:58,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:58,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:58,177 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,177 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:58,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:58,179 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:58,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:58,181 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:58,181 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:58,181 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:58,181 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:58,181 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:58,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,188 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,191 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,191 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,191 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,191 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,191 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,191 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,191 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,191 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:58,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:58,191 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,192 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:58,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,192 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:58,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720898252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,292 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:58,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:58,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:58,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:58,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:58,296 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:58,296 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:58,296 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:58,296 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:58,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,302 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,302 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:58,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:58,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:58,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:58,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:58,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:58,305 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,305 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:58,305 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:58,305 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:58,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:58,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,306 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:58,306 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:58,309 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,309 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:58,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29f65bd8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:58,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:58,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:58,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:58,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:58,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:58,320 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:58,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:58,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:58,323 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:58,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:58,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:58,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:58,324 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:58,327 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,327 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,327 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,328 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,330 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,335 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:58,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:58,334 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,335 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:58,335 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:58,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:58,435 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:58,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:58,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:58,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:58,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:58,444 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:58,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:58,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:58,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:58,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:58,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720898457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,492 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/1aa5d7f9434e40afb942bb18059de7ed 2023-04-05 18:53:58,498 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,498 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,498 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,498 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:58,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:58,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:58,500 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:58,500 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:58,500 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:58,500 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:58,501 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:58,501 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,501 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:58,502 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:58,502 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:58,503 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/576a7f33fa0a4c02a75b1f92fdc59dd2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/576a7f33fa0a4c02a75b1f92fdc59dd2 2023-04-05 18:53:58,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/576a7f33fa0a4c02a75b1f92fdc59dd2, entries=150, sequenceid=691, filesize=11.9 K 2023-04-05 18:53:58,513 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/2c1df05889044d84bee5951ee0a37eaa as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2c1df05889044d84bee5951ee0a37eaa 2023-04-05 18:53:58,525 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2c1df05889044d84bee5951ee0a37eaa, entries=150, sequenceid=691, filesize=11.9 K 2023-04-05 18:53:58,525 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-05 18:53:58,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/1aa5d7f9434e40afb942bb18059de7ed as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1aa5d7f9434e40afb942bb18059de7ed 2023-04-05 18:53:58,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1aa5d7f9434e40afb942bb18059de7ed, entries=150, sequenceid=691, filesize=11.9 K 2023-04-05 18:53:58,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for be0aa03376eb89131764dd34306ebfb5 in 498ms, sequenceid=691, compaction requested=true 2023-04-05 18:53:58,534 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:58,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:58,534 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:53:58,534 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:53:58,535 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:58,535 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:58,537 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:53:58,537 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/B is initiating minor compaction (all files) 2023-04-05 18:53:58,537 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/B in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,537 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/09f5c85718044a10a2da7dcb6d6c7402, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6efddd297f0540c2ac35f752daff5279, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/9f7504c6d68a4ea69ab9ec91f2477bf3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/756862c5b03e4dc1a304efdac9812089, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2c1df05889044d84bee5951ee0a37eaa] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=60.4 K 2023-04-05 18:53:58,537 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:53:58,537 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/A is initiating minor compaction (all files) 2023-04-05 18:53:58,537 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 09f5c85718044a10a2da7dcb6d6c7402, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1680720835479 2023-04-05 18:53:58,537 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/A in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,538 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ff2e525bf60e46efbd1d60856502d2cc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/a381afb03ef64e6faec9a58e38cf41a1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/37aa12cd6eae4eed96511de819fef2c8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/68d59a7c1a9347b5acf3db8c9f363efe, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/576a7f33fa0a4c02a75b1f92fdc59dd2] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=60.4 K 2023-04-05 18:53:58,538 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 6efddd297f0540c2ac35f752daff5279, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1680720835869 2023-04-05 18:53:58,538 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting ff2e525bf60e46efbd1d60856502d2cc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1680720835479 2023-04-05 18:53:58,538 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 9f7504c6d68a4ea69ab9ec91f2477bf3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1680720837008 2023-04-05 18:53:58,538 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting a381afb03ef64e6faec9a58e38cf41a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1680720835869 2023-04-05 18:53:58,539 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 756862c5b03e4dc1a304efdac9812089, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1680720837377 2023-04-05 18:53:58,539 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 37aa12cd6eae4eed96511de819fef2c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1680720837008 2023-04-05 18:53:58,539 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 68d59a7c1a9347b5acf3db8c9f363efe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1680720837377 2023-04-05 18:53:58,539 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 2c1df05889044d84bee5951ee0a37eaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680720838024 2023-04-05 18:53:58,541 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 576a7f33fa0a4c02a75b1f92fdc59dd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680720838024 2023-04-05 18:53:58,558 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#A#compaction#95 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:58,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:58,563 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 18:53:58,563 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:58,563 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:58,564 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:58,564 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:58,564 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:58,564 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69bfe80f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:58,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:58,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,573 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#B#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:58,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:58,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720898589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,605 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:58,605 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,605 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:58,613 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:58,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:58,626 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:58,626 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:58,626 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:58,626 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:58,627 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:58,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,635 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,636 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/3219420262c4461793330aa7c79538c4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/3219420262c4461793330aa7c79538c4 2023-04-05 18:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,638 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,640 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/cb92abba19d94b1a9d15d0d6aba518db 2023-04-05 18:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,643 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/a2d202ada4a446cdaa85b839d4955199 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a2d202ada4a446cdaa85b839d4955199 2023-04-05 18:53:58,647 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,647 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,648 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,648 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,648 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,648 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,648 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,648 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,648 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,648 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:58,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:58,648 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/A of be0aa03376eb89131764dd34306ebfb5 into 3219420262c4461793330aa7c79538c4(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:58,648 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:58,648 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:58,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:58,648 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:58,649 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/A, priority=11, startTime=1680720838534; duration=0sec 2023-04-05 18:53:58,649 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:53:58,649 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:53:58,652 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:53:58,652 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): be0aa03376eb89131764dd34306ebfb5/C is initiating minor compaction (all files) 2023-04-05 18:53:58,652 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of be0aa03376eb89131764dd34306ebfb5/C in TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,652 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/88d7209620d94220b22f6b03c96d14d3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6a8ef3adc1a44853892700c5cec8333b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1edc0dc85d7e4bb89930c789476dcda8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a11498c44338400488172dee365b3427, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1aa5d7f9434e40afb942bb18059de7ed] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp, totalSize=60.4 K 2023-04-05 18:53:58,653 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 88d7209620d94220b22f6b03c96d14d3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1680720835479 2023-04-05 18:53:58,654 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6a8ef3adc1a44853892700c5cec8333b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1680720835869 2023-04-05 18:53:58,654 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 1edc0dc85d7e4bb89930c789476dcda8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1680720837008 2023-04-05 18:53:58,655 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting a11498c44338400488172dee365b3427, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1680720837377 2023-04-05 18:53:58,655 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 1aa5d7f9434e40afb942bb18059de7ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680720838024 2023-04-05 18:53:58,656 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/B of be0aa03376eb89131764dd34306ebfb5 into a2d202ada4a446cdaa85b839d4955199(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:58,656 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:58,656 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/B, priority=11, startTime=1680720838534; duration=0sec 2023-04-05 18:53:58,656 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:58,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/a758210bfbf44a4fa15537480b8e13d0 2023-04-05 18:53:58,669 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): be0aa03376eb89131764dd34306ebfb5#C#compaction#99 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:53:58,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:58,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720898694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,749 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:58,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:58,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:58,752 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:58,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:58,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:58,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:58,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,753 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,753 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,754 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,754 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:58,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:58,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:58,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:58,755 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,755 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:58,755 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:58,755 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:58,756 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:58,757 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,757 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d1700b3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:58,759 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:58,759 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,760 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:58,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:58,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:58,760 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:58,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:58,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720898759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:58,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:58,765 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:58,765 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:58,765 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:58,765 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:58,765 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:58,766 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,767 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,767 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,767 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,769 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,771 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,771 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,771 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,771 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,771 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:58,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,771 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:58,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:58,771 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,771 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,771 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,771 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,771 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:58,771 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:58,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,871 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:58,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:58,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:58,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:58,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:58,875 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:58,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:58,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:58,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:58,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,876 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:58,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:58,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,882 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,882 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:58,884 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28ad4d90[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:58,886 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:58,886 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,891 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,892 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:58,892 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:58,892 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,892 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:58,892 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:58,893 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:58,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:58,895 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:58,895 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:58,895 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:58,895 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:58,895 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:58,897 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,897 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,897 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:58,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,898 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:58,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720898897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,900 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,902 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,902 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,902 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:58,902 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:58,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:58,902 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:58,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:58,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:58,902 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:58,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:58,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:58,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:58,903 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:58,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:58,903 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:59,003 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:59,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:59,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:59,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:59,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:59,008 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:59,008 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:59,008 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:59,008 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:59,009 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,009 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:59,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,010 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:59,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:59,011 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,011 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:59,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:59,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:59,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:59,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:59,013 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:59,013 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:59,013 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:59,013 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:53:59,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:53:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:59,015 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:59,018 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:59,018 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@331efbc6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:53:59,020 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:53:59,020 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:59,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:59,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:59,027 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:53:59,027 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:53:59,027 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:53:59,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:53:59,029 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:53:59,029 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:59,029 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:53:59,029 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:59,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:59,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:53:59,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:53:59,034 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:53:59,034 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:53:59,034 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:53:59,034 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:53:59,034 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:53:59,036 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,036 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:53:59,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,036 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:59,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:53:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,039 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:53:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:59,041 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:53:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:53:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,043 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:59,043 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:53:59,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:59,043 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,043 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:59,043 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:59,043 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:53:59,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:53:59,043 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:59,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:53:59,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:53:59,044 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:53:59,044 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:53:59,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:53:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:53:59,044 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:53:59,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42946 deadline: 1680720899064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,068 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:53:59,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42912 deadline: 1680720899068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,070 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:53:59,084 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/419b93ff6046443389c7b76a5ae49169 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/419b93ff6046443389c7b76a5ae49169 2023-04-05 18:53:59,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/1ea718f6eac74d54ad65ae1c4b39960a 2023-04-05 18:53:59,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42904 deadline: 1680720899090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,093 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8200 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:53:59,094 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in be0aa03376eb89131764dd34306ebfb5/C of be0aa03376eb89131764dd34306ebfb5 into 419b93ff6046443389c7b76a5ae49169(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:53:59,094 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:59,094 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5., storeName=be0aa03376eb89131764dd34306ebfb5/C, priority=11, startTime=1680720838535; duration=0sec 2023-04-05 18:53:59,094 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:53:59,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/cb92abba19d94b1a9d15d0d6aba518db as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/cb92abba19d94b1a9d15d0d6aba518db 2023-04-05 18:53:59,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/cb92abba19d94b1a9d15d0d6aba518db, entries=200, sequenceid=728, filesize=14.3 K 2023-04-05 18:53:59,104 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/a758210bfbf44a4fa15537480b8e13d0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a758210bfbf44a4fa15537480b8e13d0 2023-04-05 18:53:59,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a758210bfbf44a4fa15537480b8e13d0, entries=150, sequenceid=728, filesize=11.9 K 2023-04-05 18:53:59,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/1ea718f6eac74d54ad65ae1c4b39960a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1ea718f6eac74d54ad65ae1c4b39960a 2023-04-05 18:53:59,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1ea718f6eac74d54ad65ae1c4b39960a, entries=150, sequenceid=728, filesize=11.9 K 2023-04-05 18:53:59,121 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for be0aa03376eb89131764dd34306ebfb5 in 558ms, sequenceid=728, compaction requested=false 2023-04-05 18:53:59,121 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:53:59,144 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:53:59,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:53:59,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:53:59,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:53:59,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:53:59,150 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:53:59,150 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:59,150 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:53:59,150 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:53:59,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:59,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:53:59,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:53:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:53:59,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:53:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:59,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:53:59,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:53:59,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:53:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:53:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:53:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:53:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:53:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:53:59,156 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:53:59,156 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:53:59,156 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 18:53:59,156 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:53:59,157 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:59,157 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:53:59,157 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:59,157 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:53:59,157 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:53:59,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:53:59,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:53:59,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720839236, max timestamp=1680720839257Num uniques 859; 2023-04-05 18:53:59,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720839236, max timestamp=1680720839257Num uniques 859; 2023-04-05 18:53:59,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:53:59,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720839236, max timestamp=1680720839257Num uniques 859; 2023-04-05 18:53:59,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:53:59,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:53:59,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720899288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720899289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720899394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720899394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,587 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/18e15ca34af3452494ac46978080a2e8 2023-04-05 18:53:59,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720899598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720899598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,600 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/03770af80ce94e008132af159edfa3f1 2023-04-05 18:53:59,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:53:59,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720899901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:53:59,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720899902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,013 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/f8b87a2213be4867be7380874ecd9495 2023-04-05 18:54:00,028 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/18e15ca34af3452494ac46978080a2e8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/18e15ca34af3452494ac46978080a2e8 2023-04-05 18:54:00,036 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/18e15ca34af3452494ac46978080a2e8, entries=150, sequenceid=740, filesize=11.9 K 2023-04-05 18:54:00,038 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/03770af80ce94e008132af159edfa3f1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/03770af80ce94e008132af159edfa3f1 2023-04-05 18:54:00,044 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/03770af80ce94e008132af159edfa3f1, entries=150, sequenceid=740, filesize=11.9 K 2023-04-05 18:54:00,045 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/f8b87a2213be4867be7380874ecd9495 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/f8b87a2213be4867be7380874ecd9495 2023-04-05 18:54:00,050 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/f8b87a2213be4867be7380874ecd9495, entries=150, sequenceid=740, filesize=11.9 K 2023-04-05 18:54:00,051 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for be0aa03376eb89131764dd34306ebfb5 in 895ms, sequenceid=740, compaction requested=true 2023-04-05 18:54:00,051 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:54:00,051 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:54:00,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:00,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:00,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:00,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:00,053 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,053 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:00,055 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:00,055 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:00,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:00,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f7823a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:00,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:00,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:00,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:00,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:00,060 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,060 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:00,061 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:00,061 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,061 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:00,061 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:00,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:00,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:00,066 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:00,066 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:00,066 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:00,066 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:00,066 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:00,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:00,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:00,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:00,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:00,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:00,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:00,073 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:00,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,076 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:00,076 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:00,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:00,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:00,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:00,076 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:00,076 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:00,076 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,076 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:00,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:00,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:00,076 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:00,076 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:00,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:00,077 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:00,177 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:00,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:00,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:00,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:00,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:00,180 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:00,180 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:00,180 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:00,180 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:00,181 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:00,181 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:00,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,182 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:00,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:00,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:00,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:00,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:00,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:00,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:00,183 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:00,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:54:00,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:54:00,184 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 18:54:00,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:54:00,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:00,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:54:00,185 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:00,185 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:54:00,185 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:00,193 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=777 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/d10c11525e36488b9dbb58cd75410d63 2023-04-05 18:54:00,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:54:00,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:54:00,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:00,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720900415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720900416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:00,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720900521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:00,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720900521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,605 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=777 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/b1666b935ac84a8cab2075e57b00e415 2023-04-05 18:54:00,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:00,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720900724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:00,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:00,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720900724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,017 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=777 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/b31409ee2f4e4f718cefe66322a37b1a 2023-04-05 18:54:01,022 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/d10c11525e36488b9dbb58cd75410d63 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/d10c11525e36488b9dbb58cd75410d63 2023-04-05 18:54:01,026 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/d10c11525e36488b9dbb58cd75410d63, entries=150, sequenceid=777, filesize=11.9 K 2023-04-05 18:54:01,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:01,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42922 deadline: 1680720901027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,027 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/b1666b935ac84a8cab2075e57b00e415 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/b1666b935ac84a8cab2075e57b00e415 2023-04-05 18:54:01,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:01,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42958 deadline: 1680720901028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,034 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/b1666b935ac84a8cab2075e57b00e415, entries=150, sequenceid=777, filesize=11.9 K 2023-04-05 18:54:01,034 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/b31409ee2f4e4f718cefe66322a37b1a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/b31409ee2f4e4f718cefe66322a37b1a 2023-04-05 18:54:01,039 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/b31409ee2f4e4f718cefe66322a37b1a, entries=150, sequenceid=777, filesize=11.9 K 2023-04-05 18:54:01,040 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for be0aa03376eb89131764dd34306ebfb5 in 856ms, sequenceid=777, compaction requested=true 2023-04-05 18:54:01,040 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:54:01,040 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:54:01,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:01,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:01,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:01,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:01,041 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,042 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:01,042 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:01,042 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50901caf[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:01,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:01,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:01,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:01,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:01,045 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:01,045 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:01,045 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:01,045 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,045 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:01,051 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:01,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:01,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:01,054 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:01,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:01,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:01,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:01,054 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:01,055 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:01,055 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:01,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:01,055 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:01,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:01,056 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:01,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:01,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:01,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:01,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:01,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:01,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:01,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:01,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:01,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:01,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:01,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:01,059 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:01,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:01,060 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:01,060 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:01,060 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:01,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:01,160 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:01,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:01,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:01,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:01,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:01,164 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:01,164 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:01,164 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:01,164 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:01,164 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:01,164 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:01,165 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,165 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:01,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:01,166 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:01,166 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:01,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:01,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:01,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:01,167 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:54:01,167 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. started... 2023-04-05 18:54:01,168 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 18:54:01,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:54:01,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:01,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:54:01,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:01,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:54:01,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:01,299 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fab6dfa to 127.0.0.1:49755 2023-04-05 18:54:01,299 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:01,299 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:49755 2023-04-05 18:54:01,299 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:01,300 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2879e2e1 to 127.0.0.1:49755 2023-04-05 18:54:01,300 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:01,302 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bd6fc59 to 127.0.0.1:49755 2023-04-05 18:54:01,302 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:01,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:54:01,531 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. as already flushing 2023-04-05 18:54:01,531 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:49755 2023-04-05 18:54:01,531 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:01,535 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:49755 2023-04-05 18:54:01,535 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:01,577 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/d15f9bdfdce34e12bf4d37de0e92db87 2023-04-05 18:54:01,988 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/9d6563551bfc48daaa4c34727f1556ea 2023-04-05 18:54:02,398 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/43c7f794fe2744e48876f5894aecb1ac 2023-04-05 18:54:02,404 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/d15f9bdfdce34e12bf4d37de0e92db87 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/d15f9bdfdce34e12bf4d37de0e92db87 2023-04-05 18:54:02,407 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/d15f9bdfdce34e12bf4d37de0e92db87, entries=150, sequenceid=786, filesize=11.9 K 2023-04-05 18:54:02,408 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/9d6563551bfc48daaa4c34727f1556ea as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/9d6563551bfc48daaa4c34727f1556ea 2023-04-05 18:54:02,412 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/9d6563551bfc48daaa4c34727f1556ea, entries=150, sequenceid=786, filesize=11.9 K 2023-04-05 18:54:02,413 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/43c7f794fe2744e48876f5894aecb1ac as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/43c7f794fe2744e48876f5894aecb1ac 2023-04-05 18:54:02,417 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/43c7f794fe2744e48876f5894aecb1ac, entries=150, sequenceid=786, filesize=11.9 K 2023-04-05 18:54:02,418 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=13.42 KB/13740 for be0aa03376eb89131764dd34306ebfb5 in 1250ms, sequenceid=786, compaction requested=true 2023-04-05 18:54:02,418 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:54:02,418 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:02,418 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:02,420 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:02,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:02,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12130ae0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:02,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:02,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:02,423 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:02,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:02,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:02,423 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,423 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:02,425 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:02,427 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:02,427 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:02,427 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:02,427 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:02,427 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:02,428 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:02,428 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:02,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:02,428 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:02,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:02,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:02,430 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,431 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:02,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:02,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:02,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:02,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:02,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:02,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:02,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:02,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:02,432 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:02,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:02,432 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:02,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:02,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:02,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:02,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:02,532 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:02,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:09,108 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46192d17 to 127.0.0.1:49755 2023-04-05 18:54:09,108 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:09,141 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49a45b6d to 127.0.0.1:49755 2023-04-05 18:54:09,141 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:09,179 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e91d243 to 127.0.0.1:49755 2023-04-05 18:54:09,179 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:09,179 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 18:54:09,179 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 156 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 202 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 4607 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 4413 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1993 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5978 rows 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1993 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5976 rows 2023-04-05 18:54:09,180 INFO [Listener at localhost/43705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 18:54:09,180 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4281aac2 to 127.0.0.1:49755 2023-04-05 18:54:09,180 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:09,186 INFO [Listener at localhost/43705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 18:54:09,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 18:54:09,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:09,201 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720849201"}]},"ts":"1680720849201"} 2023-04-05 18:54:09,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 18:54:09,203 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 18:54:09,204 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 18:54:09,206 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=be0aa03376eb89131764dd34306ebfb5, UNASSIGN}] 2023-04-05 18:54:09,208 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=be0aa03376eb89131764dd34306ebfb5, UNASSIGN 2023-04-05 18:54:09,209 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=be0aa03376eb89131764dd34306ebfb5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:09,209 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720849208"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720849208"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720849208"}]},"ts":"1680720849208"} 2023-04-05 18:54:09,211 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:54:09,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 18:54:09,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:54:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing be0aa03376eb89131764dd34306ebfb5, disabling compactions & flushes 2023-04-05 18:54:09,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:54:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:54:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. after waiting 0 ms 2023-04-05 18:54:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:54:09,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing be0aa03376eb89131764dd34306ebfb5 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 18:54:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=A 2023-04-05 18:54:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=B 2023-04-05 18:54:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK be0aa03376eb89131764dd34306ebfb5, store=C 2023-04-05 18:54:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:09,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 18:54:09,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/0966005c57064e36af33b932b634fbde 2023-04-05 18:54:09,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 18:54:10,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/0a31d47069cb41098e474156c49987ef 2023-04-05 18:54:10,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 18:54:10,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/442276401e3148e8b6392962bf3f9f4e 2023-04-05 18:54:10,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/A/0966005c57064e36af33b932b634fbde as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/0966005c57064e36af33b932b634fbde 2023-04-05 18:54:10,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/0966005c57064e36af33b932b634fbde, entries=150, sequenceid=794, filesize=11.9 K 2023-04-05 18:54:10,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/B/0a31d47069cb41098e474156c49987ef as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0a31d47069cb41098e474156c49987ef 2023-04-05 18:54:10,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0a31d47069cb41098e474156c49987ef, entries=150, sequenceid=794, filesize=11.9 K 2023-04-05 18:54:10,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/.tmp/C/442276401e3148e8b6392962bf3f9f4e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/442276401e3148e8b6392962bf3f9f4e 2023-04-05 18:54:10,643 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/442276401e3148e8b6392962bf3f9f4e, entries=150, sequenceid=794, filesize=11.9 K 2023-04-05 18:54:10,644 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for be0aa03376eb89131764dd34306ebfb5 in 1273ms, sequenceid=794, compaction requested=true 2023-04-05 18:54:10,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/e3574efb46e94b9e9aad23a41b6276a1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/15b961f2c13e449898ee0d0464449d11, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ae16fae4ab1343248da2beb73866b6fa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/490a41f41cf646a48cd087ac64387c56, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/59cb9a000c444bbe8c390edbb1a72eec, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/3163486607794186b204f79d2e1b51ba, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5cb68cb5055f45f491d4ebcfb33afaa3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/38650f87ad75468e93b3bbc95999d3eb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5a30b03c7d6746789244cd16c6a0a6ae, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/b4e7a6cbdd8440b882de174703cec221, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/c2abdd42e49b42e3b74a738f812202fc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/7f15aae80622403e8010ba5f6a331745, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/0a281cdc445d4aba9d7dfbaa5aa2de03, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/db31589c9b9f4527aad8f88a4cada90d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/344b59a558f14d55af661384ca05350b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/2b1b8abcd52f448c81cf66fe79ef4406, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/14f6cd9d94654480b0aecc56aef769a5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/393e15aefed34766aba24d1cbb24e6c3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/91b6a9c1532b4f62868a93c35df88477, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6dc3df3f272e4228a0d9a63db5b57b25, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6d8e9fb06bcf41b9997c013f63ac528e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/7bf139a3074b4a1eb8ab4f2ce42d2fe4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/16d6965726c3490c8b56135f2f0e0203, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/bc659b99038d449bb88fa192841fee26, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f0bcf8d69320402e8ec3e76ea6e1314e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f3dc423e3a464737b9a545907a0ba959, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/cc7a56d6a2ff4034ac553504965602fa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ff2e525bf60e46efbd1d60856502d2cc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/a381afb03ef64e6faec9a58e38cf41a1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/37aa12cd6eae4eed96511de819fef2c8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/68d59a7c1a9347b5acf3db8c9f363efe, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/576a7f33fa0a4c02a75b1f92fdc59dd2] to archive 2023-04-05 18:54:10,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:54:10,678 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/e3574efb46e94b9e9aad23a41b6276a1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/e3574efb46e94b9e9aad23a41b6276a1 2023-04-05 18:54:10,680 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/15b961f2c13e449898ee0d0464449d11 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/15b961f2c13e449898ee0d0464449d11 2023-04-05 18:54:10,681 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ae16fae4ab1343248da2beb73866b6fa to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ae16fae4ab1343248da2beb73866b6fa 2023-04-05 18:54:10,683 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/490a41f41cf646a48cd087ac64387c56 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/490a41f41cf646a48cd087ac64387c56 2023-04-05 18:54:10,684 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/59cb9a000c444bbe8c390edbb1a72eec to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/59cb9a000c444bbe8c390edbb1a72eec 2023-04-05 18:54:10,686 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/3163486607794186b204f79d2e1b51ba to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/3163486607794186b204f79d2e1b51ba 2023-04-05 18:54:10,688 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5cb68cb5055f45f491d4ebcfb33afaa3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5cb68cb5055f45f491d4ebcfb33afaa3 2023-04-05 18:54:10,690 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/38650f87ad75468e93b3bbc95999d3eb to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/38650f87ad75468e93b3bbc95999d3eb 2023-04-05 18:54:10,693 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5a30b03c7d6746789244cd16c6a0a6ae to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/5a30b03c7d6746789244cd16c6a0a6ae 2023-04-05 18:54:10,694 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/b4e7a6cbdd8440b882de174703cec221 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/b4e7a6cbdd8440b882de174703cec221 2023-04-05 18:54:10,696 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/c2abdd42e49b42e3b74a738f812202fc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/c2abdd42e49b42e3b74a738f812202fc 2023-04-05 18:54:10,697 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/7f15aae80622403e8010ba5f6a331745 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/7f15aae80622403e8010ba5f6a331745 2023-04-05 18:54:10,699 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/0a281cdc445d4aba9d7dfbaa5aa2de03 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/0a281cdc445d4aba9d7dfbaa5aa2de03 2023-04-05 18:54:10,700 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/db31589c9b9f4527aad8f88a4cada90d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/db31589c9b9f4527aad8f88a4cada90d 2023-04-05 18:54:10,702 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/344b59a558f14d55af661384ca05350b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/344b59a558f14d55af661384ca05350b 2023-04-05 18:54:10,703 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/2b1b8abcd52f448c81cf66fe79ef4406 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/2b1b8abcd52f448c81cf66fe79ef4406 2023-04-05 18:54:10,705 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/14f6cd9d94654480b0aecc56aef769a5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/14f6cd9d94654480b0aecc56aef769a5 2023-04-05 18:54:10,706 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/393e15aefed34766aba24d1cbb24e6c3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/393e15aefed34766aba24d1cbb24e6c3 2023-04-05 18:54:10,708 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/91b6a9c1532b4f62868a93c35df88477 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/91b6a9c1532b4f62868a93c35df88477 2023-04-05 18:54:10,709 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6dc3df3f272e4228a0d9a63db5b57b25 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6dc3df3f272e4228a0d9a63db5b57b25 2023-04-05 18:54:10,710 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6d8e9fb06bcf41b9997c013f63ac528e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/6d8e9fb06bcf41b9997c013f63ac528e 2023-04-05 18:54:10,712 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/7bf139a3074b4a1eb8ab4f2ce42d2fe4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/7bf139a3074b4a1eb8ab4f2ce42d2fe4 2023-04-05 18:54:10,714 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/16d6965726c3490c8b56135f2f0e0203 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/16d6965726c3490c8b56135f2f0e0203 2023-04-05 18:54:10,715 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/bc659b99038d449bb88fa192841fee26 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/bc659b99038d449bb88fa192841fee26 2023-04-05 18:54:10,716 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f0bcf8d69320402e8ec3e76ea6e1314e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f0bcf8d69320402e8ec3e76ea6e1314e 2023-04-05 18:54:10,717 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f3dc423e3a464737b9a545907a0ba959 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/f3dc423e3a464737b9a545907a0ba959 2023-04-05 18:54:10,718 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/cc7a56d6a2ff4034ac553504965602fa to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/cc7a56d6a2ff4034ac553504965602fa 2023-04-05 18:54:10,720 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ff2e525bf60e46efbd1d60856502d2cc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/ff2e525bf60e46efbd1d60856502d2cc 2023-04-05 18:54:10,721 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/a381afb03ef64e6faec9a58e38cf41a1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/a381afb03ef64e6faec9a58e38cf41a1 2023-04-05 18:54:10,722 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/37aa12cd6eae4eed96511de819fef2c8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/37aa12cd6eae4eed96511de819fef2c8 2023-04-05 18:54:10,723 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/68d59a7c1a9347b5acf3db8c9f363efe to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/68d59a7c1a9347b5acf3db8c9f363efe 2023-04-05 18:54:10,725 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/576a7f33fa0a4c02a75b1f92fdc59dd2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/576a7f33fa0a4c02a75b1f92fdc59dd2 2023-04-05 18:54:10,749 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/88a430b42cf649d8a4239283a2824245, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/c75d942efdf243caa616063d18ff79f5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/dc9c73a4a31d411794f6c5eb8654728a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4b2c981f2fba455a9a6857a476c7de9c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0b79b3bab3a54817bbe579e42334bb96, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6a8f0ece9add4d73a6fe4721ff24fa3c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/715e5ce06f9d45af85f1e49e30bc97f7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a737367978d044878d16041b6407be23, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1519d9f1feb2450daa80ba868df60901, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/31a6aea2a6714c1f9570c2322c55b55b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5c975630cb554c73ba96a91d1a3a2ac9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0cc30d20932f4f72ac10181e867ebf6d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/193826160b284e3ca71efc82fe04d131, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/7fc30bdaceb54a82a95cef1cfddd790a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/210e6e99220f4dfc8b12438a91e13cdc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1c01290c378c4644b1dab9d22982d3b2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/243eb79f795b403a9e4f0771ecd9642c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2ea9de25f9474eb694736ea7297adf45, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4d9a36b7ac124288966c02af96cb9204, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/50a568afa43c4fb3a490c5d21aa1ba4b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/eb282a6a691b4efcbb627f57cea32829, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/ccdaaf2804c8488aa32fe4f554866b8e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/f6b1bd92f5a84222ab8ce1a1c4a06b47, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/27975b32d5a848e6bf11b695e5ad9dd6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/29c0c196eeae4ef9954b5ea6191a322d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5a2fb3c40f50492ab3134f3c9234a808, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/09f5c85718044a10a2da7dcb6d6c7402, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5ff036f18b234e77b8ad94dc2a04c6ac, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6efddd297f0540c2ac35f752daff5279, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/9f7504c6d68a4ea69ab9ec91f2477bf3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/756862c5b03e4dc1a304efdac9812089, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2c1df05889044d84bee5951ee0a37eaa] to archive 2023-04-05 18:54:10,750 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:54:10,752 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/88a430b42cf649d8a4239283a2824245 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/88a430b42cf649d8a4239283a2824245 2023-04-05 18:54:10,754 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/c75d942efdf243caa616063d18ff79f5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/c75d942efdf243caa616063d18ff79f5 2023-04-05 18:54:10,755 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/dc9c73a4a31d411794f6c5eb8654728a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/dc9c73a4a31d411794f6c5eb8654728a 2023-04-05 18:54:10,756 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4b2c981f2fba455a9a6857a476c7de9c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4b2c981f2fba455a9a6857a476c7de9c 2023-04-05 18:54:10,758 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0b79b3bab3a54817bbe579e42334bb96 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0b79b3bab3a54817bbe579e42334bb96 2023-04-05 18:54:10,759 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6a8f0ece9add4d73a6fe4721ff24fa3c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6a8f0ece9add4d73a6fe4721ff24fa3c 2023-04-05 18:54:10,760 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/715e5ce06f9d45af85f1e49e30bc97f7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/715e5ce06f9d45af85f1e49e30bc97f7 2023-04-05 18:54:10,761 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a737367978d044878d16041b6407be23 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a737367978d044878d16041b6407be23 2023-04-05 18:54:10,762 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1519d9f1feb2450daa80ba868df60901 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1519d9f1feb2450daa80ba868df60901 2023-04-05 18:54:10,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/31a6aea2a6714c1f9570c2322c55b55b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/31a6aea2a6714c1f9570c2322c55b55b 2023-04-05 18:54:10,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5c975630cb554c73ba96a91d1a3a2ac9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5c975630cb554c73ba96a91d1a3a2ac9 2023-04-05 18:54:10,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0cc30d20932f4f72ac10181e867ebf6d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0cc30d20932f4f72ac10181e867ebf6d 2023-04-05 18:54:10,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/193826160b284e3ca71efc82fe04d131 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/193826160b284e3ca71efc82fe04d131 2023-04-05 18:54:10,767 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/7fc30bdaceb54a82a95cef1cfddd790a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/7fc30bdaceb54a82a95cef1cfddd790a 2023-04-05 18:54:10,768 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/210e6e99220f4dfc8b12438a91e13cdc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/210e6e99220f4dfc8b12438a91e13cdc 2023-04-05 18:54:10,769 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1c01290c378c4644b1dab9d22982d3b2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/1c01290c378c4644b1dab9d22982d3b2 2023-04-05 18:54:10,770 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/243eb79f795b403a9e4f0771ecd9642c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/243eb79f795b403a9e4f0771ecd9642c 2023-04-05 18:54:10,771 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2ea9de25f9474eb694736ea7297adf45 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2ea9de25f9474eb694736ea7297adf45 2023-04-05 18:54:10,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4d9a36b7ac124288966c02af96cb9204 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/4d9a36b7ac124288966c02af96cb9204 2023-04-05 18:54:10,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/50a568afa43c4fb3a490c5d21aa1ba4b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/50a568afa43c4fb3a490c5d21aa1ba4b 2023-04-05 18:54:10,774 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/eb282a6a691b4efcbb627f57cea32829 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/eb282a6a691b4efcbb627f57cea32829 2023-04-05 18:54:10,776 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/ccdaaf2804c8488aa32fe4f554866b8e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/ccdaaf2804c8488aa32fe4f554866b8e 2023-04-05 18:54:10,777 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/f6b1bd92f5a84222ab8ce1a1c4a06b47 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/f6b1bd92f5a84222ab8ce1a1c4a06b47 2023-04-05 18:54:10,778 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/27975b32d5a848e6bf11b695e5ad9dd6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/27975b32d5a848e6bf11b695e5ad9dd6 2023-04-05 18:54:10,779 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/29c0c196eeae4ef9954b5ea6191a322d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/29c0c196eeae4ef9954b5ea6191a322d 2023-04-05 18:54:10,780 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5a2fb3c40f50492ab3134f3c9234a808 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5a2fb3c40f50492ab3134f3c9234a808 2023-04-05 18:54:10,781 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/09f5c85718044a10a2da7dcb6d6c7402 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/09f5c85718044a10a2da7dcb6d6c7402 2023-04-05 18:54:10,782 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5ff036f18b234e77b8ad94dc2a04c6ac to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/5ff036f18b234e77b8ad94dc2a04c6ac 2023-04-05 18:54:10,783 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6efddd297f0540c2ac35f752daff5279 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/6efddd297f0540c2ac35f752daff5279 2023-04-05 18:54:10,784 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/9f7504c6d68a4ea69ab9ec91f2477bf3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/9f7504c6d68a4ea69ab9ec91f2477bf3 2023-04-05 18:54:10,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/756862c5b03e4dc1a304efdac9812089 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/756862c5b03e4dc1a304efdac9812089 2023-04-05 18:54:10,786 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2c1df05889044d84bee5951ee0a37eaa to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/2c1df05889044d84bee5951ee0a37eaa 2023-04-05 18:54:10,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3b741579c73d4381b3aa71013d10d592, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bdcb800ee1754159bf3f1a46f931af3b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/81b03c1c76de4ecfa2744ba71a9502a6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/97888120c78245409cdd29bc22f5a4ff, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/276c7ec8623b48188af47a3173ed7a3f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7d819f334c8e46ae88e1a7efb3e8a865, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/89f9f05466ab41dcbd901f662059fd11, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a4f524bb1e95416a902128a9f0d8e25b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ef422707c214a228e1ec9ac731e08ab, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3eccb99c5ff94614bf7fa6695119db85, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/26858bd6beeb4938b49dbe03abecd25a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/0bc8f03c3eed45a19fb5450c7dd147bc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6aac9f7f11f54724b0bc2c7bb0de60ae, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/03113f63815046188034c20774a11760, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/b3dbd554bf77452fbbacdbeb5d7706d2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/847802b9cb4b4af6845304c323bb862a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ba937be173045d1a83c93fc11ce7f52, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/638f0c0e9beb42d9aa09aba7416e474e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a46c44e642a5447a96f87d9c53761d09, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/cd9178152a4b49e89d9e3ff6e8592fee, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bd52f11a2b2f4b16a773501319f57b90, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/631c71bade9c4ebd93ec88bccd8bbac6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/137d75df309f4825a9bdd9c494d539af, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/ba9937ebe57d4834b10d7597587efcd0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/73a7f4e4ea8943b48b5abdb145b3afc3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/88d7209620d94220b22f6b03c96d14d3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/171453c3ad98449ca1a45e9edaf81e28, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6a8ef3adc1a44853892700c5cec8333b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1edc0dc85d7e4bb89930c789476dcda8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a11498c44338400488172dee365b3427, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1aa5d7f9434e40afb942bb18059de7ed] to archive 2023-04-05 18:54:10,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:54:10,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3b741579c73d4381b3aa71013d10d592 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3b741579c73d4381b3aa71013d10d592 2023-04-05 18:54:10,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bdcb800ee1754159bf3f1a46f931af3b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bdcb800ee1754159bf3f1a46f931af3b 2023-04-05 18:54:10,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/81b03c1c76de4ecfa2744ba71a9502a6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/81b03c1c76de4ecfa2744ba71a9502a6 2023-04-05 18:54:10,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/97888120c78245409cdd29bc22f5a4ff to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/97888120c78245409cdd29bc22f5a4ff 2023-04-05 18:54:10,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/276c7ec8623b48188af47a3173ed7a3f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/276c7ec8623b48188af47a3173ed7a3f 2023-04-05 18:54:10,797 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7d819f334c8e46ae88e1a7efb3e8a865 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7d819f334c8e46ae88e1a7efb3e8a865 2023-04-05 18:54:10,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/89f9f05466ab41dcbd901f662059fd11 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/89f9f05466ab41dcbd901f662059fd11 2023-04-05 18:54:10,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a4f524bb1e95416a902128a9f0d8e25b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a4f524bb1e95416a902128a9f0d8e25b 2023-04-05 18:54:10,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ef422707c214a228e1ec9ac731e08ab to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ef422707c214a228e1ec9ac731e08ab 2023-04-05 18:54:10,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3eccb99c5ff94614bf7fa6695119db85 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/3eccb99c5ff94614bf7fa6695119db85 2023-04-05 18:54:10,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/26858bd6beeb4938b49dbe03abecd25a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/26858bd6beeb4938b49dbe03abecd25a 2023-04-05 18:54:10,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/0bc8f03c3eed45a19fb5450c7dd147bc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/0bc8f03c3eed45a19fb5450c7dd147bc 2023-04-05 18:54:10,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6aac9f7f11f54724b0bc2c7bb0de60ae to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6aac9f7f11f54724b0bc2c7bb0de60ae 2023-04-05 18:54:10,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/03113f63815046188034c20774a11760 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/03113f63815046188034c20774a11760 2023-04-05 18:54:10,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/b3dbd554bf77452fbbacdbeb5d7706d2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/b3dbd554bf77452fbbacdbeb5d7706d2 2023-04-05 18:54:10,809 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/847802b9cb4b4af6845304c323bb862a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/847802b9cb4b4af6845304c323bb862a 2023-04-05 18:54:10,809 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ba937be173045d1a83c93fc11ce7f52 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/7ba937be173045d1a83c93fc11ce7f52 2023-04-05 18:54:10,810 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/638f0c0e9beb42d9aa09aba7416e474e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/638f0c0e9beb42d9aa09aba7416e474e 2023-04-05 18:54:10,811 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a46c44e642a5447a96f87d9c53761d09 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a46c44e642a5447a96f87d9c53761d09 2023-04-05 18:54:10,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/cd9178152a4b49e89d9e3ff6e8592fee to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/cd9178152a4b49e89d9e3ff6e8592fee 2023-04-05 18:54:10,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bd52f11a2b2f4b16a773501319f57b90 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/bd52f11a2b2f4b16a773501319f57b90 2023-04-05 18:54:10,814 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/631c71bade9c4ebd93ec88bccd8bbac6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/631c71bade9c4ebd93ec88bccd8bbac6 2023-04-05 18:54:10,815 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/137d75df309f4825a9bdd9c494d539af to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/137d75df309f4825a9bdd9c494d539af 2023-04-05 18:54:10,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/ba9937ebe57d4834b10d7597587efcd0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/ba9937ebe57d4834b10d7597587efcd0 2023-04-05 18:54:10,817 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/73a7f4e4ea8943b48b5abdb145b3afc3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/73a7f4e4ea8943b48b5abdb145b3afc3 2023-04-05 18:54:10,818 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/88d7209620d94220b22f6b03c96d14d3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/88d7209620d94220b22f6b03c96d14d3 2023-04-05 18:54:10,819 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/171453c3ad98449ca1a45e9edaf81e28 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/171453c3ad98449ca1a45e9edaf81e28 2023-04-05 18:54:10,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6a8ef3adc1a44853892700c5cec8333b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/6a8ef3adc1a44853892700c5cec8333b 2023-04-05 18:54:10,821 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1edc0dc85d7e4bb89930c789476dcda8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1edc0dc85d7e4bb89930c789476dcda8 2023-04-05 18:54:10,822 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a11498c44338400488172dee365b3427 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/a11498c44338400488172dee365b3427 2023-04-05 18:54:10,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1aa5d7f9434e40afb942bb18059de7ed to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1aa5d7f9434e40afb942bb18059de7ed 2023-04-05 18:54:10,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/recovered.edits/797.seqid, newMaxSeqId=797, maxSeqId=1 2023-04-05 18:54:10,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5. 2023-04-05 18:54:10,829 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for be0aa03376eb89131764dd34306ebfb5: 2023-04-05 18:54:10,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:54:10,831 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=be0aa03376eb89131764dd34306ebfb5, regionState=CLOSED 2023-04-05 18:54:10,831 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720850831"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720850831"}]},"ts":"1680720850831"} 2023-04-05 18:54:10,834 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-05 18:54:10,834 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure be0aa03376eb89131764dd34306ebfb5, server=jenkins-hbase4.apache.org,33489,1680720816595 in 1.6210 sec 2023-04-05 18:54:10,835 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-05 18:54:10,835 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=be0aa03376eb89131764dd34306ebfb5, UNASSIGN in 1.6280 sec 2023-04-05 18:54:10,836 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720850836"}]},"ts":"1680720850836"} 2023-04-05 18:54:10,837 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 18:54:10,839 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 18:54:10,841 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6480 sec 2023-04-05 18:54:11,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 18:54:11,307 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-05 18:54:11,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 18:54:11,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:11,319 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:11,320 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:11,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-05 18:54:11,325 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:54:11,327 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/recovered.edits] 2023-04-05 18:54:11,330 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/0966005c57064e36af33b932b634fbde to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/0966005c57064e36af33b932b634fbde 2023-04-05 18:54:11,332 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/18e15ca34af3452494ac46978080a2e8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/18e15ca34af3452494ac46978080a2e8 2023-04-05 18:54:11,333 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/3219420262c4461793330aa7c79538c4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/3219420262c4461793330aa7c79538c4 2023-04-05 18:54:11,334 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/cb92abba19d94b1a9d15d0d6aba518db to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/cb92abba19d94b1a9d15d0d6aba518db 2023-04-05 18:54:11,335 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/d10c11525e36488b9dbb58cd75410d63 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/d10c11525e36488b9dbb58cd75410d63 2023-04-05 18:54:11,337 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/d15f9bdfdce34e12bf4d37de0e92db87 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/A/d15f9bdfdce34e12bf4d37de0e92db87 2023-04-05 18:54:11,339 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/03770af80ce94e008132af159edfa3f1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/03770af80ce94e008132af159edfa3f1 2023-04-05 18:54:11,340 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0a31d47069cb41098e474156c49987ef to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/0a31d47069cb41098e474156c49987ef 2023-04-05 18:54:11,341 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/9d6563551bfc48daaa4c34727f1556ea to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/9d6563551bfc48daaa4c34727f1556ea 2023-04-05 18:54:11,342 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a2d202ada4a446cdaa85b839d4955199 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a2d202ada4a446cdaa85b839d4955199 2023-04-05 18:54:11,344 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a758210bfbf44a4fa15537480b8e13d0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/a758210bfbf44a4fa15537480b8e13d0 2023-04-05 18:54:11,345 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/b1666b935ac84a8cab2075e57b00e415 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/B/b1666b935ac84a8cab2075e57b00e415 2023-04-05 18:54:11,348 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1ea718f6eac74d54ad65ae1c4b39960a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/1ea718f6eac74d54ad65ae1c4b39960a 2023-04-05 18:54:11,349 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/419b93ff6046443389c7b76a5ae49169 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/419b93ff6046443389c7b76a5ae49169 2023-04-05 18:54:11,350 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/43c7f794fe2744e48876f5894aecb1ac to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/43c7f794fe2744e48876f5894aecb1ac 2023-04-05 18:54:11,351 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/442276401e3148e8b6392962bf3f9f4e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/442276401e3148e8b6392962bf3f9f4e 2023-04-05 18:54:11,353 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/b31409ee2f4e4f718cefe66322a37b1a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/b31409ee2f4e4f718cefe66322a37b1a 2023-04-05 18:54:11,354 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/f8b87a2213be4867be7380874ecd9495 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/C/f8b87a2213be4867be7380874ecd9495 2023-04-05 18:54:11,357 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/recovered.edits/797.seqid to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5/recovered.edits/797.seqid 2023-04-05 18:54:11,358 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/be0aa03376eb89131764dd34306ebfb5 2023-04-05 18:54:11,358 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:54:11,360 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:11,381 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 18:54:11,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-05 18:54:11,427 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 18:54:11,429 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:11,429 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 18:54:11,429 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680720851429"}]},"ts":"9223372036854775807"} 2023-04-05 18:54:11,431 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 18:54:11,431 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => be0aa03376eb89131764dd34306ebfb5, NAME => 'TestAcidGuarantees,,1680720820031.be0aa03376eb89131764dd34306ebfb5.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 18:54:11,431 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 18:54:11,431 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680720851431"}]},"ts":"9223372036854775807"} 2023-04-05 18:54:11,433 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 18:54:11,435 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:11,436 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 123 msec 2023-04-05 18:54:11,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-05 18:54:11,623 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-05 18:54:11,640 INFO [Listener at localhost/43705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 206) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5fffeea1-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_873678755_15 at /127.0.0.1:50302 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5fffeea1-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:33489-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5fffeea1-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5fffeea1-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=551 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=325 (was 297) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 170), AvailableMemoryMB=1724 (was 2317) 2023-04-05 18:54:11,650 INFO [Listener at localhost/43705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=551, MaxFileDescriptor=60000, SystemLoadAverage=325, ProcessCount=169, AvailableMemoryMB=1724 2023-04-05 18:54:11,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 18:54:11,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 18:54:11,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:11,654 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 18:54:11,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-05 18:54:11,655 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 18:54:11,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 18:54:11,656 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:11,657 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d empty. 2023-04-05 18:54:11,657 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:11,657 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:54:11,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 18:54:11,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 18:54:12,066 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 18:54:12,068 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5f2134cd982ecee57a32108ad09a562d, NAME => 'TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp 2023-04-05 18:54:12,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 18:54:12,476 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:54:12,476 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5f2134cd982ecee57a32108ad09a562d, disabling compactions & flushes 2023-04-05 18:54:12,476 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:12,476 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:12,476 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. after waiting 0 ms 2023-04-05 18:54:12,476 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:12,476 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:12,476 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:12,478 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 18:54:12,479 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720852479"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720852479"}]},"ts":"1680720852479"} 2023-04-05 18:54:12,480 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 18:54:12,481 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 18:54:12,482 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720852482"}]},"ts":"1680720852482"} 2023-04-05 18:54:12,482 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 18:54:12,485 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, ASSIGN}] 2023-04-05 18:54:12,487 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, ASSIGN 2023-04-05 18:54:12,487 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33489,1680720816595; forceNewPlan=false, retain=false 2023-04-05 18:54:12,638 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=5f2134cd982ecee57a32108ad09a562d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:12,639 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720852638"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720852638"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720852638"}]},"ts":"1680720852638"} 2023-04-05 18:54:12,640 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:54:12,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 18:54:12,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:12,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5f2134cd982ecee57a32108ad09a562d, NAME => 'TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:54:12,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:12,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:54:12,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:12,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:12,797 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:12,799 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A 2023-04-05 18:54:12,799 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A 2023-04-05 18:54:12,799 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:54:12,799 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5f2134cd982ecee57a32108ad09a562d columnFamilyName A 2023-04-05 18:54:12,800 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(310): Store=5f2134cd982ecee57a32108ad09a562d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:54:12,800 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:12,801 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B 2023-04-05 18:54:12,801 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B 2023-04-05 18:54:12,801 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:54:12,801 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5f2134cd982ecee57a32108ad09a562d columnFamilyName B 2023-04-05 18:54:12,802 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(310): Store=5f2134cd982ecee57a32108ad09a562d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:54:12,802 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:12,803 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C 2023-04-05 18:54:12,803 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C 2023-04-05 18:54:12,803 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:54:12,803 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5f2134cd982ecee57a32108ad09a562d columnFamilyName C 2023-04-05 18:54:12,804 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(310): Store=5f2134cd982ecee57a32108ad09a562d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:54:12,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:12,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:12,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:12,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:54:12,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:12,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 18:54:12,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5f2134cd982ecee57a32108ad09a562d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69229754, jitterRate=0.031603723764419556}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:54:12,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:12,811 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., pid=18, masterSystemTime=1680720852792 2023-04-05 18:54:12,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:12,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:12,812 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=5f2134cd982ecee57a32108ad09a562d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:12,813 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720852812"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680720852812"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680720852812"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680720852812"}]},"ts":"1680720852812"} 2023-04-05 18:54:12,815 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-05 18:54:12,816 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 in 174 msec 2023-04-05 18:54:12,817 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-05 18:54:12,817 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, ASSIGN in 331 msec 2023-04-05 18:54:12,818 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 18:54:12,818 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720852818"}]},"ts":"1680720852818"} 2023-04-05 18:54:12,819 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 18:54:12,822 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 18:54:12,823 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1700 sec 2023-04-05 18:54:13,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 18:54:13,760 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-05 18:54:13,761 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x324dc814 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:13,766 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a301dcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:13,768 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:13,771 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55744, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:13,773 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 18:54:13,775 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39232, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 18:54:13,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 18:54:13,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 18:54:13,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:13,800 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 18:54:13,801 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 18:54:13,801 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 18:54:13,805 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-05 18:54:13,817 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, REOPEN/MOVE}] 2023-04-05 18:54:13,818 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, REOPEN/MOVE 2023-04-05 18:54:13,818 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5f2134cd982ecee57a32108ad09a562d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:13,818 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720853818"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720853818"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720853818"}]},"ts":"1680720853818"} 2023-04-05 18:54:13,820 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:54:13,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:13,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5f2134cd982ecee57a32108ad09a562d, disabling compactions & flushes 2023-04-05 18:54:13,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:13,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:13,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. after waiting 0 ms 2023-04-05 18:54:13,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:13,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-05 18:54:13,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:13,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:13,977 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 5f2134cd982ecee57a32108ad09a562d to self. 2023-04-05 18:54:13,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:13,979 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5f2134cd982ecee57a32108ad09a562d, regionState=CLOSED 2023-04-05 18:54:13,979 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720853979"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720853979"}]},"ts":"1680720853979"} 2023-04-05 18:54:13,982 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-05 18:54:13,982 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 in 161 msec 2023-04-05 18:54:13,983 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,33489,1680720816595; forceNewPlan=false, retain=true 2023-04-05 18:54:14,133 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5f2134cd982ecee57a32108ad09a562d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,134 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720854133"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720854133"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720854133"}]},"ts":"1680720854133"} 2023-04-05 18:54:14,135 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:54:14,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:14,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5f2134cd982ecee57a32108ad09a562d, NAME => 'TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:54:14,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:54:14,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,294 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,295 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A 2023-04-05 18:54:14,295 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A 2023-04-05 18:54:14,295 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:54:14,298 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5f2134cd982ecee57a32108ad09a562d columnFamilyName A 2023-04-05 18:54:14,300 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(310): Store=5f2134cd982ecee57a32108ad09a562d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:54:14,300 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,301 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B 2023-04-05 18:54:14,301 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B 2023-04-05 18:54:14,301 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:54:14,302 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5f2134cd982ecee57a32108ad09a562d columnFamilyName B 2023-04-05 18:54:14,302 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(310): Store=5f2134cd982ecee57a32108ad09a562d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:54:14,302 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,303 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C 2023-04-05 18:54:14,303 DEBUG [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C 2023-04-05 18:54:14,303 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:54:14,304 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5f2134cd982ecee57a32108ad09a562d columnFamilyName C 2023-04-05 18:54:14,304 INFO [StoreOpener-5f2134cd982ecee57a32108ad09a562d-1] regionserver.HStore(310): Store=5f2134cd982ecee57a32108ad09a562d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:54:14,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:14,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:54:14,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5f2134cd982ecee57a32108ad09a562d; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69232482, jitterRate=0.03164437413215637}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:54:14,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:14,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., pid=23, masterSystemTime=1680720854287 2023-04-05 18:54:14,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:14,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:14,312 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5f2134cd982ecee57a32108ad09a562d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,313 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720854312"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680720854312"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680720854312"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680720854312"}]},"ts":"1680720854312"} 2023-04-05 18:54:14,316 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-05 18:54:14,316 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 in 179 msec 2023-04-05 18:54:14,317 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-05 18:54:14,317 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, REOPEN/MOVE in 499 msec 2023-04-05 18:54:14,320 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-05 18:54:14,320 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 515 msec 2023-04-05 18:54:14,322 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 537 msec 2023-04-05 18:54:14,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-05 18:54:14,330 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x55cbb115 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:14,335 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1aa5eef5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:14,336 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:14,340 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:14,340 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:14,345 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:14,345 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:14,349 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:14,350 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:14,353 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:14,353 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:14,357 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:14,357 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:14,361 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:14,361 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x1a7b66a3 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:14,364 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2efdb02f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:14,365 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x42d325f4 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:14,368 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dc1d73, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:14,373 DEBUG [hconnection-0x7d48ceb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:14,373 DEBUG [hconnection-0x6415d427-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:14,373 DEBUG [hconnection-0x2065fa01-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:14,375 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55760, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:14,375 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55752, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:14,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:14,376 DEBUG [hconnection-0x73fed2f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:14,378 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55772, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:14,385 DEBUG [hconnection-0x509f0b2e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:14,386 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55764, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:14,385 DEBUG [hconnection-0x76506782-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:14,387 DEBUG [hconnection-0x482ba3ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:14,389 DEBUG [hconnection-0xbdcca44-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:14,389 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55806, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:14,390 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:14,394 DEBUG [hconnection-0x4714035f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:14,394 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55790, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:14,395 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55820, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:14,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:14,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:14,403 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:14,404 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:14,404 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:14,404 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:14,405 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,405 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,405 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55836, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:14,407 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:14,407 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:14,408 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,408 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,413 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:14,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:14,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:14,418 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:14,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:14,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:14,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:14,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:14,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:14,420 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:14,420 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:14,420 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:14,420 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:14,420 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:14,420 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:14,420 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:14,420 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:14,420 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,420 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:14,420 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:14,424 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:14,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:14,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:14,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@569713af[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:14,427 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:14,427 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,428 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:14,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:14,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:14,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:14,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:14,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:14,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:14,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:14,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:14,437 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:14,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:14,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:14,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:14,437 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:14,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:14,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:14,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:14,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,442 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:14,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:14,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:14,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:14,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:14,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:14,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:14,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,446 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,446 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:14,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:14,446 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:14,446 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:14,482 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405de94920695e14a21b75b72090ef3d42e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405de94920695e14a21b75b72090ef3d42e 2023-04-05 18:54:14,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720914474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720914479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720914481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720914484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720914484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,510 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/381d324829664cb59286483366ad4b5f 2023-04-05 18:54:14,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/eab9407073ce49fdb0af5c42b3f69f14 2023-04-05 18:54:14,546 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:14,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:14,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:14,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/1a5e850f4fdc441899e318f62389f5ed 2023-04-05 18:54:14,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:14,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:14,584 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:14,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:14,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:14,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:14,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:14,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:14,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:14,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/381d324829664cb59286483366ad4b5f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/381d324829664cb59286483366ad4b5f 2023-04-05 18:54:14,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720914587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720914589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720914590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720914596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720914599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:14,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:14,603 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/381d324829664cb59286483366ad4b5f, entries=150, sequenceid=17, filesize=25.1 K 2023-04-05 18:54:14,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/eab9407073ce49fdb0af5c42b3f69f14 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/eab9407073ce49fdb0af5c42b3f69f14 2023-04-05 18:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:14,604 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:14,604 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:14,604 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:14,604 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,608 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:14,608 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:14,610 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,610 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f68bd15[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:14,611 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:14,611 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,612 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:14,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:14,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:14,613 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,613 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:14,614 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:14,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:14,618 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:14,618 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:14,618 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:14,618 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:14,618 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:14,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:14,621 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:14,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,624 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,626 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/eab9407073ce49fdb0af5c42b3f69f14, entries=150, sequenceid=17, filesize=11.6 K 2023-04-05 18:54:14,626 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:14,626 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:14,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:14,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:14,626 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,626 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,626 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:14,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:14,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:14,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:14,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:14,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:14,627 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/1a5e850f4fdc441899e318f62389f5ed as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1a5e850f4fdc441899e318f62389f5ed 2023-04-05 18:54:14,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:14,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,627 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:14,627 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:14,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1a5e850f4fdc441899e318f62389f5ed, entries=150, sequenceid=17, filesize=11.6 K 2023-04-05 18:54:14,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 5f2134cd982ecee57a32108ad09a562d in 225ms, sequenceid=17, compaction requested=false 2023-04-05 18:54:14,638 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:14,727 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:14,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:14,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:14,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:14,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:14,744 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:14,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:14,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:14,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:14,746 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,746 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,748 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:14,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:14,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:14,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:14,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:14,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:14,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:14,752 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:14,752 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:14,752 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:14,752 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:14,752 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:14,752 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:14,752 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:14,753 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:54:14,753 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:14,753 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:14,753 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:14,753 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:14,753 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:14,753 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:14,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:14,802 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:14,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720914809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720914810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720914813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720914813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720914814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,838 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304050400633465c04bb98b2274567880045e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050400633465c04bb98b2274567880045e 2023-04-05 18:54:14,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720914914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720914917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720914918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720914919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:14,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:14,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720914919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720915123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720915123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720915123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720915124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720915124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,272 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/abb3727124de48e983a1275a69895be6 2023-04-05 18:54:15,328 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/94e5db6517354152b1a6bf77ab7bcd37 2023-04-05 18:54:15,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720915428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720915429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720915430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720915430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720915432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,754 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/e8dda066b9104e9f90f3ef294d4f41a0 2023-04-05 18:54:15,760 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/abb3727124de48e983a1275a69895be6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/abb3727124de48e983a1275a69895be6 2023-04-05 18:54:15,765 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/abb3727124de48e983a1275a69895be6, entries=150, sequenceid=37, filesize=25.1 K 2023-04-05 18:54:15,766 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/94e5db6517354152b1a6bf77ab7bcd37 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/94e5db6517354152b1a6bf77ab7bcd37 2023-04-05 18:54:15,775 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/94e5db6517354152b1a6bf77ab7bcd37, entries=150, sequenceid=37, filesize=11.6 K 2023-04-05 18:54:15,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/e8dda066b9104e9f90f3ef294d4f41a0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e8dda066b9104e9f90f3ef294d4f41a0 2023-04-05 18:54:15,781 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e8dda066b9104e9f90f3ef294d4f41a0, entries=150, sequenceid=37, filesize=11.6 K 2023-04-05 18:54:15,782 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5f2134cd982ecee57a32108ad09a562d in 1030ms, sequenceid=37, compaction requested=false 2023-04-05 18:54:15,782 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:15,782 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:15,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:15,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:15,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:15,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:15,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:15,784 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:15,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:15,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:15,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:15,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:15,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:15,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4db533[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:15,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:15,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:15,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:15,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:15,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:15,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:15,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:15,788 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,788 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:15,790 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:15,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:15,794 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:15,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:15,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:15,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:15,794 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:15,797 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:15,797 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:15,797 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:15,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:15,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:15,798 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:15,800 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:15,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:15,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:15,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:15,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:15,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:15,804 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:15,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:15,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:15,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:15,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:15,804 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:15,804 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:15,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:15,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:15,904 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:15,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:15,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:15,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:15,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:15,909 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:15,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:15,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:15,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:15,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:15,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:15,911 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:15,911 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:15,912 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,912 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:15,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:15,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:15,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:15,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:15,922 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:15,922 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:15,922 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:54:15,927 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:15,927 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:15,927 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:15,927 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:15,927 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:15,927 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:15,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:15,932 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:15,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4442) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4381) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4857) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4851) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4847) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3159) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720915952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720915956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720855944, max timestamp=1680720855957Num uniques 859; 2023-04-05 18:54:15,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:15,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720855944, max timestamp=1680720855957Num uniques 859; 2023-04-05 18:54:15,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720915957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:15,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720855944, max timestamp=1680720855957Num uniques 859; 2023-04-05 18:54:15,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:15,965 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304055173f58faa7a49b2b9646cd3a79b79a4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055173f58faa7a49b2b9646cd3a79b79a4 2023-04-05 18:54:15,982 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/d9dfecf59dec45448dbadf026bbd339f 2023-04-05 18:54:15,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720915992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:15,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:15,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720915996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,009 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/4a68f82768224e099bfd7e23fa80eec6 2023-04-05 18:54:16,025 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/3c674b3e402f4297b9926c71e34245d5 2023-04-05 18:54:16,031 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/d9dfecf59dec45448dbadf026bbd339f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/d9dfecf59dec45448dbadf026bbd339f 2023-04-05 18:54:16,037 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/d9dfecf59dec45448dbadf026bbd339f, entries=150, sequenceid=49, filesize=25.1 K 2023-04-05 18:54:16,038 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/4a68f82768224e099bfd7e23fa80eec6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/4a68f82768224e099bfd7e23fa80eec6 2023-04-05 18:54:16,044 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/4a68f82768224e099bfd7e23fa80eec6, entries=150, sequenceid=49, filesize=11.6 K 2023-04-05 18:54:16,045 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/3c674b3e402f4297b9926c71e34245d5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/3c674b3e402f4297b9926c71e34245d5 2023-04-05 18:54:16,050 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/3c674b3e402f4297b9926c71e34245d5, entries=150, sequenceid=49, filesize=11.6 K 2023-04-05 18:54:16,051 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 5f2134cd982ecee57a32108ad09a562d in 129ms, sequenceid=49, compaction requested=true 2023-04-05 18:54:16,051 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:16,052 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:16,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:16,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,056 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:16,056 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:16,057 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:16,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:16,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1503e830[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:16,061 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:16,061 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,063 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:16,063 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:16,063 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:16,064 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,064 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:16,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:16,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:16,065 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,068 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-05 18:54:16,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:16,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:16,069 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:16,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:16,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:16,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:16,069 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:16,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:16,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:16,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:16,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:16,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:16,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:16,074 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,075 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,075 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:16,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:16,083 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:16,083 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:16,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720916087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720916088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720916090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720916098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720916099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,183 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:16,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:16,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:16,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:16,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:16,187 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:16,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:16,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:16,188 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,188 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,190 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,190 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,190 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:16,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:16,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:16,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:16,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:16,192 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,192 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:16,192 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:16,192 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:16,194 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,195 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,195 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:16,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b679b54[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:16,196 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:16,196 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,197 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,197 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:16,197 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:16,197 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:16,197 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,198 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:16,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720916194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,198 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720916194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:16,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:16,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:16,201 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:16,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:16,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:16,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:16,201 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:16,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720916197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,204 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,204 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,204 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:16,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:16,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,209 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,209 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:16,209 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:16,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720916304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720916306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,309 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:16,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:16,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:16,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:16,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:16,313 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:16,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:16,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:16,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,314 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:16,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:16,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:16,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:16,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:16,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:16,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:16,316 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,316 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:16,316 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:16,316 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:16,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:16,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:16,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:16,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:16,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@716d1138[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:16,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:16,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:16,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:16,324 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:16,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,324 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:16,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:16,327 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:16,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:16,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:16,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:16,327 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:16,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,329 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,332 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:16,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:16,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,334 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:16,334 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:16,334 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,375 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 18:54:16,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720916401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720916401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720916403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,435 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:16,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:16,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:16,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:16,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:16,439 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:16,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:16,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:16,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,441 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,442 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,442 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,442 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:16,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:16,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:16,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:16,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:16,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:16,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:16,443 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,443 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:16,444 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:16,444 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:16,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:16,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:16,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:16,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:16,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:16,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37bd60b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:16,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,451 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:16,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,451 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:16,451 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:16,451 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:16,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:16,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,453 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:16,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:16,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:16,457 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:16,457 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:16,457 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:16,457 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:16,457 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:16,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,462 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,466 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:16,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:16,466 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:16,466 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:16,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:16,468 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,468 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,468 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,468 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,468 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,468 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,468 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,527 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d9f90a2aec52410c98b13b641b73ec68 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d9f90a2aec52410c98b13b641b73ec68 2023-04-05 18:54:16,567 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:16,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:16,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:16,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:16,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:16,571 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:16,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:16,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:16,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,577 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:16,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:16,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:16,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:16,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:16,581 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:16,581 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:16,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:16,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:16,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:16,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:16,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b3ec3c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:16,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:16,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,587 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:16,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:16,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:16,589 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:16,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:16,590 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:16,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:16,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:16,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:16,590 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:16,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,594 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,595 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:16,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:16,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:16,600 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,600 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:16,600 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:16,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720916608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720916609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,700 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:16,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:16,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:16,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:16,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:16,704 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:16,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:16,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:16,705 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,705 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,706 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,706 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:16,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720916704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720916705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:16,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:16,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:16,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:16,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720916708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:16,712 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,713 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,713 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:16,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec96b65[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:16,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:16,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,718 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:16,718 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:16,718 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:16,718 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,718 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:16,719 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:16,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:16,723 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:16,723 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:16,723 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:16,723 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:16,724 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:16,725 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,725 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,725 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,726 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,728 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,730 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,730 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,730 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,730 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,730 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:16,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,730 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,730 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,731 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,731 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,731 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,731 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:16,731 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:16,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,831 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:16,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:16,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:16,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:16,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:16,835 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:16,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:16,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:16,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,838 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,838 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:16,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:16,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:16,840 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,840 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:16,840 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:16,840 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:16,842 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,842 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,842 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,842 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:16,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57344755[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:16,844 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:16,844 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,845 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:16,845 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:16,845 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:16,845 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,845 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:16,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,846 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:16,846 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,846 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:16,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:16,849 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:16,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:16,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:16,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:16,849 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:16,851 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,851 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,852 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,852 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,854 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,858 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,858 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,859 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:16,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:16,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:16,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,859 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,859 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:16,859 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:16,933 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=87, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/6f95db6cc0f745c699b4dfed2c7e0b9a 2023-04-05 18:54:16,955 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/9caf8f817694420c923f7ed39ccf5f6f 2023-04-05 18:54:16,959 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:16,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:16,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:16,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:16,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:16,963 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:16,963 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,963 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:16,963 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:16,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,967 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,967 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:16,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:16,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:16,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:16,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:16,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:16,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:16,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:16,972 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,972 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:16,972 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:16,972 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:16,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:16,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:16,973 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,973 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:16,973 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:16,975 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,975 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,975 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,975 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:16,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:16,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa21836[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:16,978 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:16,978 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:16,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:16,979 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,980 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:16,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,981 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:16,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:16,983 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:16,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:16,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:16,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:16,984 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:16,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,987 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:16,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,987 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,991 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:16,991 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:16,991 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:16,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:16,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:16,991 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:16,992 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:16,992 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:16,991 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,992 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:16,992 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:16,992 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,992 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:16,992 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:16,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:16,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:17,092 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:17,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:17,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:17,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:17,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:17,096 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:17,096 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:17,096 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:17,096 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:17,096 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,096 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:17,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:17,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,098 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,098 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:17,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:17,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:17,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:17,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:17,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:17,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:17,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:17,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:17,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:17,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:17,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:17,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce6d752[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:17,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:17,103 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:17,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:17,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:17,105 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,105 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:17,106 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:17,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:17,109 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:17,109 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:17,109 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:17,109 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:17,109 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:17,110 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,110 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:17,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,110 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:17,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:17,111 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,113 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:17,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:17,115 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:17,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:17,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:17,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:17,115 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:17,115 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:17,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:17,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:17,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:17,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720917118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:17,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720917118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:17,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720917209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:17,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720917212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:17,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720917213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,216 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:17,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:17,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:17,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:17,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:17,219 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:17,219 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:17,219 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:17,219 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:17,220 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,220 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,222 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:17,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:17,223 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,223 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:17,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:17,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:17,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:17,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:17,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:17,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:17,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:17,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:17,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:17,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:17,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:17,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:17,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:17,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,225 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:17,225 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:17,226 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,226 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:17,226 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,226 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:17,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:17,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:17,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:17,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19f105e8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:17,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:17,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,229 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,229 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:17,229 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:17,229 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:17,229 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,229 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:17,230 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:17,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:17,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:17,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:17,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:17,233 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:17,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:17,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:17,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:17,233 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:17,235 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,235 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:17,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,235 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:17,235 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:17,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:17,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,240 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:17,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:17,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:17,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,243 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:17,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:17,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:17,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:17,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:17,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,243 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:17,243 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:17,343 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:17,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:17,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:17,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:17,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:17,352 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:17,352 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:17,352 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:17,352 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:17,352 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,352 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,354 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,354 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,354 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:17,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:17,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:17,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:17,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:17,359 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:17,359 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:17,359 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:17,360 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:17,360 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:17,360 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:17,360 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:17,365 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:17,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:17,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:17,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c8af76c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:17,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:17,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,368 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:17,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:17,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:17,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:17,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,369 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:17,369 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:17,369 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:17,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:17,373 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:17,373 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:17,373 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:17,373 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:17,373 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:17,375 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,375 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,375 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:17,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:17,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:17,375 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:17,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:17,376 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/89dec585cfc84e47b10bc2503871ec2b 2023-04-05 18:54:17,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,380 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:17,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/6f95db6cc0f745c699b4dfed2c7e0b9a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6f95db6cc0f745c699b4dfed2c7e0b9a 2023-04-05 18:54:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:17,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:17,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:17,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:17,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:17,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:17,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:17,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:17,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:17,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:17,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:17,385 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:17,386 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:17,388 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6f95db6cc0f745c699b4dfed2c7e0b9a, entries=150, sequenceid=87, filesize=25.1 K 2023-04-05 18:54:17,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/9caf8f817694420c923f7ed39ccf5f6f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/9caf8f817694420c923f7ed39ccf5f6f 2023-04-05 18:54:17,394 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/9caf8f817694420c923f7ed39ccf5f6f, entries=150, sequenceid=87, filesize=11.6 K 2023-04-05 18:54:17,394 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/89dec585cfc84e47b10bc2503871ec2b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/89dec585cfc84e47b10bc2503871ec2b 2023-04-05 18:54:17,399 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/89dec585cfc84e47b10bc2503871ec2b, entries=150, sequenceid=87, filesize=11.6 K 2023-04-05 18:54:17,400 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 5f2134cd982ecee57a32108ad09a562d in 1332ms, sequenceid=87, compaction requested=true 2023-04-05 18:54:17,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:17,400 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:17,400 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:54:17,400 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:54:17,400 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:17,400 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:17,401 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 102984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:54:17,402 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:54:17,402 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/A is initiating minor compaction (all files) 2023-04-05 18:54:17,402 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/B is initiating minor compaction (all files) 2023-04-05 18:54:17,402 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/A in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:17,402 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/B in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:17,402 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/381d324829664cb59286483366ad4b5f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/abb3727124de48e983a1275a69895be6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/d9dfecf59dec45448dbadf026bbd339f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6f95db6cc0f745c699b4dfed2c7e0b9a] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=100.6 K 2023-04-05 18:54:17,402 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/eab9407073ce49fdb0af5c42b3f69f14, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/94e5db6517354152b1a6bf77ab7bcd37, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/4a68f82768224e099bfd7e23fa80eec6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/9caf8f817694420c923f7ed39ccf5f6f] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=46.4 K 2023-04-05 18:54:17,402 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 381d324829664cb59286483366ad4b5f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680720854410 2023-04-05 18:54:17,402 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting eab9407073ce49fdb0af5c42b3f69f14, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680720854410 2023-04-05 18:54:17,404 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting abb3727124de48e983a1275a69895be6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680720854456 2023-04-05 18:54:17,404 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 94e5db6517354152b1a6bf77ab7bcd37, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680720854456 2023-04-05 18:54:17,404 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting d9dfecf59dec45448dbadf026bbd339f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720854799 2023-04-05 18:54:17,404 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4a68f82768224e099bfd7e23fa80eec6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720854799 2023-04-05 18:54:17,404 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6f95db6cc0f745c699b4dfed2c7e0b9a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680720855995 2023-04-05 18:54:17,405 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 9caf8f817694420c923f7ed39ccf5f6f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680720855995 2023-04-05 18:54:17,415 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#B#compaction#126 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:17,421 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#A#compaction#125 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:17,439 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/1b7ead8654e64fb9b1fdd3acf94d0ec8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1b7ead8654e64fb9b1fdd3acf94d0ec8 2023-04-05 18:54:17,446 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/B of 5f2134cd982ecee57a32108ad09a562d into 1b7ead8654e64fb9b1fdd3acf94d0ec8(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:17,446 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:17,446 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/B, priority=12, startTime=1680720857400; duration=0sec 2023-04-05 18:54:17,446 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:17,446 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:54:17,449 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:54:17,449 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/C is initiating minor compaction (all files) 2023-04-05 18:54:17,449 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/C in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:17,450 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1a5e850f4fdc441899e318f62389f5ed, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e8dda066b9104e9f90f3ef294d4f41a0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/3c674b3e402f4297b9926c71e34245d5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/89dec585cfc84e47b10bc2503871ec2b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=46.4 K 2023-04-05 18:54:17,450 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1a5e850f4fdc441899e318f62389f5ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680720854410 2023-04-05 18:54:17,450 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e8dda066b9104e9f90f3ef294d4f41a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680720854456 2023-04-05 18:54:17,451 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 3c674b3e402f4297b9926c71e34245d5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720854799 2023-04-05 18:54:17,451 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 89dec585cfc84e47b10bc2503871ec2b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680720855995 2023-04-05 18:54:17,472 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#C#compaction#127 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:17,486 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:17,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:17,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:17,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:17,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:17,493 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:17,493 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:17,493 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:17,493 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:17,494 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,494 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,495 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:17,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:17,495 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,496 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:17,496 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:17,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:17,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:17,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:17,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:17,500 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:17,500 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:17,500 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:17,519 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:17,519 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:17,519 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:17,519 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:17,519 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 18:54:17,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:17,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:17,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:17,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:17,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:17,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:17,529 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/a85fca9fea22493a8c97df850c9cd3c3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a85fca9fea22493a8c97df850c9cd3c3 2023-04-05 18:54:17,535 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/A of 5f2134cd982ecee57a32108ad09a562d into a85fca9fea22493a8c97df850c9cd3c3(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:17,535 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:17,535 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/A, priority=12, startTime=1680720857400; duration=0sec 2023-04-05 18:54:17,536 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:17,542 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304051b0e52125136480eb708f7bfec2006a2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051b0e52125136480eb708f7bfec2006a2 2023-04-05 18:54:17,555 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/41f7d25ca9654c8e875f13e1afb87311 2023-04-05 18:54:17,922 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/bd007bbd0d194acdb8a5a5e24280d0db as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/bd007bbd0d194acdb8a5a5e24280d0db 2023-04-05 18:54:17,930 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/C of 5f2134cd982ecee57a32108ad09a562d into bd007bbd0d194acdb8a5a5e24280d0db(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:17,930 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:17,930 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/C, priority=12, startTime=1680720857400; duration=0sec 2023-04-05 18:54:17,931 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:17,974 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/3a2a12e828974259b517a3ab1e603a99 2023-04-05 18:54:18,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:18,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:18,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680720858155, max timestamp=1680720858165Num uniques 859; 2023-04-05 18:54:18,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 18:54:18,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720918174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720918176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720918216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720918219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720918221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720918277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720918279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,399 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/5e41949dcd954c52b636c8d145afc77e 2023-04-05 18:54:18,405 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/41f7d25ca9654c8e875f13e1afb87311 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/41f7d25ca9654c8e875f13e1afb87311 2023-04-05 18:54:18,413 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/41f7d25ca9654c8e875f13e1afb87311, entries=150, sequenceid=97, filesize=25.1 K 2023-04-05 18:54:18,414 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/3a2a12e828974259b517a3ab1e603a99 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/3a2a12e828974259b517a3ab1e603a99 2023-04-05 18:54:18,419 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/3a2a12e828974259b517a3ab1e603a99, entries=150, sequenceid=97, filesize=11.6 K 2023-04-05 18:54:18,420 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/5e41949dcd954c52b636c8d145afc77e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/5e41949dcd954c52b636c8d145afc77e 2023-04-05 18:54:18,427 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/5e41949dcd954c52b636c8d145afc77e, entries=150, sequenceid=97, filesize=11.6 K 2023-04-05 18:54:18,428 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=136.42 KB/139690 for 5f2134cd982ecee57a32108ad09a562d in 909ms, sequenceid=97, compaction requested=false 2023-04-05 18:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,430 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76732163[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:18,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:18,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,434 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:18,434 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,434 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:18,435 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:18,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:18,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:18,438 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:18,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:18,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:18,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:18,438 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:18,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:18,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:18,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:18,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:18,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:18,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:18,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:18,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:18,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,444 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:18,444 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:18,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:18,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:18,483 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=143.13 KB heapSize=416.80 KB 2023-04-05 18:54:18,484 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:18,484 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:18,484 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:18,484 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:18,484 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:18,484 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:18,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720918490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720918491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,544 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:18,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:18,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:18,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:18,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:18,548 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:18,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:18,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:18,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:18,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:18,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:18,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:18,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:18,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:18,552 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:18,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:18,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:18,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:18,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:18,553 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:18,553 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:18,553 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:18,553 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:18,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:18,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:18,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:18,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:18,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0534b8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:18,557 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:18,557 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,558 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,558 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:18,558 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,558 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:18,558 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:18,558 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:18,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,559 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,559 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:18,562 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:18,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:18,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:18,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:18,562 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:18,565 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,565 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,565 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:18,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,566 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,568 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:18,568 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:18,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:18,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:18,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:18,568 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:18,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:18,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:18,569 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:18,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,569 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:18,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720918598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720918598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,669 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:18,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:18,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:18,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:18,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:18,673 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:18,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:18,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:18,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:18,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,674 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,674 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:18,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:18,674 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,674 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:18,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:18,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:18,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:18,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:18,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:18,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:18,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:18,676 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:18,676 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:18,676 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:18,676 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:18,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:18,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:18,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:18,676 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:18,678 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcaa495[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:18,679 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:18,680 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,682 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:18,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:18,686 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:18,688 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:18,688 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:18,688 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:18,688 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:18,688 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:18,690 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,690 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,690 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:18,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,691 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,692 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:18,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:18,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:18,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:18,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:18,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:18,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:18,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:18,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,696 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:18,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,697 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,697 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,696 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:18,797 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:18,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:18,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:18,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:18,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:18,800 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:18,800 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:18,800 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:18,801 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:18,801 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,801 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,802 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,802 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:18,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:18,802 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,802 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720918802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:18,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:18,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:18,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:18,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:18,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:18,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:18,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:18,804 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:18,804 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:18,804 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:18,804 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:18,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:18,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:18,805 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:18,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720918804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:18,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76400bd7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:18,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:18,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,809 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,809 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:18,809 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:18,809 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:18,809 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,809 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:18,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:18,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:18,814 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:18,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:18,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:18,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:18,814 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:18,815 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,815 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,816 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:18,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,816 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,817 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,819 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:18,819 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:18,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:18,819 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:18,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:18,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:18,819 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:18,819 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,820 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:18,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,820 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:18,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:18,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:18,905 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040594dbb50dd02844c1ae7a97798687abd3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040594dbb50dd02844c1ae7a97798687abd3 2023-04-05 18:54:18,911 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=127, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/c1360495e21d450094dd52f9304f2a0d 2023-04-05 18:54:18,920 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:18,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:18,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:18,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:18,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:18,934 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:18,934 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:18,934 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:18,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:18,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,936 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,936 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:18,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:18,936 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,937 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:18,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:18,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:18,938 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:18,938 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:18,938 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:18,938 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:18,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:18,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:18,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:18,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:18,941 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,941 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,941 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,941 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:18,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19d75445[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:18,944 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:18,944 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,945 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,945 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:18,945 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:18,945 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:18,945 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,945 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:18,947 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:18,950 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:18,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:18,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:18,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:18,951 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:18,952 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,952 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:18,952 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:18,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:18,955 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:18,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:18,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:18,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:18,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:18,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:18,959 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:18,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:18,959 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:18,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:18,959 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:18,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:18,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,059 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:19,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:19,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:19,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:19,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:19,063 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:19,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:19,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:19,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,064 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,064 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,065 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,065 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:19,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:19,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:19,066 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,066 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:19,066 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:19,066 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:19,068 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,068 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,068 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,068 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:19,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c020333[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:19,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:19,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,072 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:19,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:19,073 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:19,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:19,077 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:19,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:19,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:19,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:19,077 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:19,081 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,081 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,081 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,081 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,086 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,086 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:19,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:19,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:19,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,086 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,087 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:19,087 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:19,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:19,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720919105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:19,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720919108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,187 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:19,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:19,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:19,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:19,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:19,191 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:19,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:19,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:19,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,192 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,193 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:19,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:19,194 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,194 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:19,194 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:19,194 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:19,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:19,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f4ec2a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:19,197 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:19,197 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,198 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:19,198 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:19,200 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,200 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:19,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:19,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:19,203 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:19,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:19,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:19,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:19,203 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:19,204 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,204 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,204 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,209 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:19,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:19,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,209 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:19,209 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:19,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:19,309 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:19,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:19,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:19,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:19,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:19,313 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:19,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:19,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:19,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,314 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:19,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:19,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:19,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:19,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:19,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:19,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:19,316 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,316 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:19,317 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:19,317 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:19,319 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:19,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21c4fd0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:19,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:19,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:19,324 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,324 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,324 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,324 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:19,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:19,327 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:19,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:19,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:19,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:19,327 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:19,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/ed8ee66c9c3d40019e1a6702052fd553 2023-04-05 18:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,332 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:19,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:19,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,337 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,338 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:19,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,338 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,338 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,338 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,338 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:19,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,438 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:19,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:19,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:19,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:19,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:19,441 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:19,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:19,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:19,442 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,442 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:19,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:19,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:19,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:19,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21ca56aa[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:19,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:19,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,450 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,450 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,450 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:19,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:19,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:19,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:19,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:19,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:19,455 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:19,455 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:19,455 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:19,455 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:19,455 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:19,456 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,456 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,456 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,457 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,459 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:19,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:19,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:19,465 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,465 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:19,465 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,465 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,565 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:19,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:19,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:19,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:19,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:19,569 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:19,569 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,569 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:19,569 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:19,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,572 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:19,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:19,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:19,573 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:19,573 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,573 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:19,574 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,574 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:19,574 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:19,574 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:19,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dadd23b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:19,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:19,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,579 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:19,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:19,581 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:19,583 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:19,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:19,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:19,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:19,583 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:19,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,586 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:19,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:19,591 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:19,591 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:19,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:19,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:19,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:19,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720919612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720919613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,691 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:19,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:19,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:19,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:19,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:19,694 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:19,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:19,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:19,695 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,695 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:19,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:19,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:19,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:19,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:19,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:19,698 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,698 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:19,698 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:19,698 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:19,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:19,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:19,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:19,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:19,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eebf018[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:19,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:19,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:19,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:19,705 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:19,707 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:19,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:19,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:19,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:19,708 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:19,709 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,709 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,709 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:19,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,713 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,713 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,713 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,713 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:19,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:19,713 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,713 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:19,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,713 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:19,713 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,713 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:19,713 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:19,713 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:19,713 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:19,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/dbfef999b29a4bdabff2cef56cd8b94b 2023-04-05 18:54:19,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/c1360495e21d450094dd52f9304f2a0d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/c1360495e21d450094dd52f9304f2a0d 2023-04-05 18:54:19,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/c1360495e21d450094dd52f9304f2a0d, entries=200, sequenceid=127, filesize=31.9 K 2023-04-05 18:54:19,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/ed8ee66c9c3d40019e1a6702052fd553 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ed8ee66c9c3d40019e1a6702052fd553 2023-04-05 18:54:19,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ed8ee66c9c3d40019e1a6702052fd553, entries=150, sequenceid=127, filesize=11.6 K 2023-04-05 18:54:19,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/dbfef999b29a4bdabff2cef56cd8b94b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/dbfef999b29a4bdabff2cef56cd8b94b 2023-04-05 18:54:19,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/dbfef999b29a4bdabff2cef56cd8b94b, entries=150, sequenceid=127, filesize=11.6 K 2023-04-05 18:54:19,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~416.75 KB/426752, currentSize=33.54 KB/34350 for 5f2134cd982ecee57a32108ad09a562d in 1298ms, sequenceid=127, compaction requested=true 2023-04-05 18:54:19,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:19,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:19,781 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:19,781 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:19,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:19,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:19,782 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84316 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:19,782 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:19,783 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/A is initiating minor compaction (all files) 2023-04-05 18:54:19,783 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/B is initiating minor compaction (all files) 2023-04-05 18:54:19,783 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/A in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,783 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/B in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,783 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a85fca9fea22493a8c97df850c9cd3c3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/41f7d25ca9654c8e875f13e1afb87311, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/c1360495e21d450094dd52f9304f2a0d] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=82.3 K 2023-04-05 18:54:19,783 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1b7ead8654e64fb9b1fdd3acf94d0ec8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/3a2a12e828974259b517a3ab1e603a99, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ed8ee66c9c3d40019e1a6702052fd553] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=35.0 K 2023-04-05 18:54:19,783 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting a85fca9fea22493a8c97df850c9cd3c3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680720855995 2023-04-05 18:54:19,783 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1b7ead8654e64fb9b1fdd3acf94d0ec8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680720855995 2023-04-05 18:54:19,784 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 41f7d25ca9654c8e875f13e1afb87311, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680720856085 2023-04-05 18:54:19,784 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 3a2a12e828974259b517a3ab1e603a99, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680720856085 2023-04-05 18:54:19,786 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c1360495e21d450094dd52f9304f2a0d, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680720858170 2023-04-05 18:54:19,786 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ed8ee66c9c3d40019e1a6702052fd553, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680720858170 2023-04-05 18:54:19,796 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#B#compaction#134 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:19,813 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:19,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:19,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:19,818 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#A#compaction#135 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:19,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:19,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:19,823 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:19,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:19,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:19,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,826 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:19,826 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:19,828 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:19,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:19,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:19,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:19,830 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:19,830 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:19,830 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 18:54:19,830 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:19,830 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:19,830 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:19,831 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:19,831 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:19,831 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:19,845 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304051fec2d0050944046a0332816d3e440a5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051fec2d0050944046a0332816d3e440a5 2023-04-05 18:54:20,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:20,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:20,244 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/674bbcdd9b9c4a9486350b962d90870d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/674bbcdd9b9c4a9486350b962d90870d 2023-04-05 18:54:20,255 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/B of 5f2134cd982ecee57a32108ad09a562d into 674bbcdd9b9c4a9486350b962d90870d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:20,256 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:20,256 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/B, priority=13, startTime=1680720859781; duration=0sec 2023-04-05 18:54:20,256 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:20,256 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:20,259 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:20,260 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/C is initiating minor compaction (all files) 2023-04-05 18:54:20,260 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/C in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:20,260 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/bd007bbd0d194acdb8a5a5e24280d0db, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/5e41949dcd954c52b636c8d145afc77e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/dbfef999b29a4bdabff2cef56cd8b94b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=35.0 K 2023-04-05 18:54:20,260 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting bd007bbd0d194acdb8a5a5e24280d0db, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680720855995 2023-04-05 18:54:20,261 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 5e41949dcd954c52b636c8d145afc77e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680720856085 2023-04-05 18:54:20,262 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting dbfef999b29a4bdabff2cef56cd8b94b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680720858170 2023-04-05 18:54:20,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720860249, max timestamp=1680720860262Num uniques 859; 2023-04-05 18:54:20,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720860249, max timestamp=1680720860262Num uniques 859; 2023-04-05 18:54:20,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:54:20,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720860249, max timestamp=1680720860262Num uniques 859; 2023-04-05 18:54:20,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:54:20,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:54:20,265 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=135, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/fd79a13abcce4fef97405bfe9406fd02 2023-04-05 18:54:20,279 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#C#compaction#137 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:20,281 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/5ebcfbb56cba4d60a7e42f327f6aafc7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5ebcfbb56cba4d60a7e42f327f6aafc7 2023-04-05 18:54:20,287 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/A of 5f2134cd982ecee57a32108ad09a562d into 5ebcfbb56cba4d60a7e42f327f6aafc7(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:20,288 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:20,288 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/A, priority=13, startTime=1680720859781; duration=0sec 2023-04-05 18:54:20,288 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:20,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720920296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680720860279, max timestamp=1680720860295Num uniques 1118; 2023-04-05 18:54:20,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-05 18:54:20,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720920297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680720860279, max timestamp=1680720860295Num uniques 1118; 2023-04-05 18:54:20,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-05 18:54:20,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680720860279, max timestamp=1680720860295Num uniques 1118; 2023-04-05 18:54:20,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-05 18:54:20,311 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/a614455a1b6a4425ae4f0e0d51eaf039 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a614455a1b6a4425ae4f0e0d51eaf039 2023-04-05 18:54:20,316 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/C of 5f2134cd982ecee57a32108ad09a562d into a614455a1b6a4425ae4f0e0d51eaf039(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:20,316 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:20,316 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/C, priority=13, startTime=1680720859781; duration=0sec 2023-04-05 18:54:20,316 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:20,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720920345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720920400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720920403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720920446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720920602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720920607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720920619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720920625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720920650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,698 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/77930071eefe45ae99af88b3a571b37a 2023-04-05 18:54:20,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720920905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720920912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:20,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:20,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720920952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,111 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/a80ac4372444416d817067f6c350d45d 2023-04-05 18:54:21,117 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/fd79a13abcce4fef97405bfe9406fd02 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/fd79a13abcce4fef97405bfe9406fd02 2023-04-05 18:54:21,126 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/fd79a13abcce4fef97405bfe9406fd02, entries=150, sequenceid=135, filesize=25.3 K 2023-04-05 18:54:21,131 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/77930071eefe45ae99af88b3a571b37a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/77930071eefe45ae99af88b3a571b37a 2023-04-05 18:54:21,137 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/77930071eefe45ae99af88b3a571b37a, entries=150, sequenceid=135, filesize=11.8 K 2023-04-05 18:54:21,138 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/a80ac4372444416d817067f6c350d45d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a80ac4372444416d817067f6c350d45d 2023-04-05 18:54:21,142 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a80ac4372444416d817067f6c350d45d, entries=150, sequenceid=135, filesize=11.8 K 2023-04-05 18:54:21,143 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 5f2134cd982ecee57a32108ad09a562d in 1313ms, sequenceid=135, compaction requested=false 2023-04-05 18:54:21,143 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:21,143 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:21,143 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:21,143 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:21,143 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,144 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:21,144 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:21,145 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:21,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:21,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee94cda[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:21,147 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:21,147 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,149 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,149 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:21,150 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,150 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:21,150 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:21,150 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:21,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:21,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:21,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:21,156 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:21,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:21,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:21,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:21,156 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:21,157 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:21,158 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:21,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,158 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:21,158 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,161 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,168 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,168 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:21,168 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:21,168 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,168 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,168 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,168 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,169 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:21,169 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:21,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:21,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:21,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:21,169 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:21,169 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:21,169 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:21,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:21,269 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:21,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:21,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:21,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:21,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:21,273 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:21,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:21,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:21,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:21,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:21,275 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:21,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:21,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:21,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:21,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:21,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:21,278 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:21,278 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:21,280 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-05 18:54:21,280 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:21,280 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:21,280 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:21,280 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:21,280 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:21,280 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:21,314 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405ae0948b4f6b14e82bd72a7fe29d33fd6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ae0948b4f6b14e82bd72a7fe29d33fd6 2023-04-05 18:54:21,322 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=192, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/e5a944ac0fcd4bab868c4ee0c26e53fb 2023-04-05 18:54:21,354 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/23363e235bcf4fbcaf1fea63d1838ff6 2023-04-05 18:54:21,382 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/893e22f04f734714a34af5ca89654a9c 2023-04-05 18:54:21,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/e5a944ac0fcd4bab868c4ee0c26e53fb as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/e5a944ac0fcd4bab868c4ee0c26e53fb 2023-04-05 18:54:21,394 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/e5a944ac0fcd4bab868c4ee0c26e53fb, entries=150, sequenceid=192, filesize=25.3 K 2023-04-05 18:54:21,395 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/23363e235bcf4fbcaf1fea63d1838ff6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/23363e235bcf4fbcaf1fea63d1838ff6 2023-04-05 18:54:21,399 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/23363e235bcf4fbcaf1fea63d1838ff6, entries=150, sequenceid=192, filesize=11.8 K 2023-04-05 18:54:21,400 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/893e22f04f734714a34af5ca89654a9c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/893e22f04f734714a34af5ca89654a9c 2023-04-05 18:54:21,409 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/893e22f04f734714a34af5ca89654a9c, entries=150, sequenceid=192, filesize=11.8 K 2023-04-05 18:54:21,410 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=0 B/0 for 5f2134cd982ecee57a32108ad09a562d in 132ms, sequenceid=192, compaction requested=true 2023-04-05 18:54:21,410 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:21,410 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:21,412 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,412 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:21,412 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,412 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de528ab[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:21,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:21,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:21,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:21,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:21,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:21,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:21,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:21,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:21,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:21,421 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:21,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:21,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:21,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:21,421 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:21,423 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,423 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,423 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:21,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:21,424 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:21,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:21,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,428 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:21,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:21,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:21,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:21,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:21,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:21,428 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:21,428 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,429 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:21,429 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:21,429 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,429 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,429 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:21,429 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,429 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:21,429 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:21,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:21,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:21,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:21,529 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:21,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:21,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:21,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:21,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:21,533 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:21,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:21,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:21,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:21,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:21,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,535 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:21,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:21,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:21,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:21,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:21,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:21,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:21,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:21,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:21,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:21,538 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:21,538 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:21,538 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-05 18:54:21,538 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:21,538 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:21,538 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:21,538 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:21,538 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:21,538 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:21,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:21,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:21,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1680720861548, max timestamp=1680720861600Num uniques 859; 2023-04-05 18:54:21,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1680720861548, max timestamp=1680720861600Num uniques 859; 2023-04-05 18:54:21,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-04-05 18:54:21,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-04-05 18:54:21,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1680720861548, max timestamp=1680720861600Num uniques 859; 2023-04-05 18:54:21,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-04-05 18:54:21,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:21,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720921619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:21,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720921624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:21,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720921624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:21,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720921725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:21,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720921732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:21,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720921732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:21,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720921927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:21,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720921936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:21,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720921936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:21,986 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405a300526bf29549da90f878fdd29c235a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a300526bf29549da90f878fdd29c235a 2023-04-05 18:54:22,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:22,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720922232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:22,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:22,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720922239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:22,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:22,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720922239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:22,393 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=197, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/5604ba7064494a83a4685f3a766a1a58 2023-04-05 18:54:22,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:22,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720922636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:22,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:22,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720922638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:22,642 DEBUG [Thread-885] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:54:22,642 DEBUG [Thread-883] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:54:22,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:22,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720922734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:22,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:22,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720922746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:22,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:22,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720922753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:22,805 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/297022b677ce4bc58adc5f92732c3ba1 2023-04-05 18:54:23,219 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/26d8906213d648ec87c3fd0bcfd5148c 2023-04-05 18:54:23,225 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/5604ba7064494a83a4685f3a766a1a58 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5604ba7064494a83a4685f3a766a1a58 2023-04-05 18:54:23,229 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5604ba7064494a83a4685f3a766a1a58, entries=100, sequenceid=197, filesize=18.5 K 2023-04-05 18:54:23,230 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/297022b677ce4bc58adc5f92732c3ba1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/297022b677ce4bc58adc5f92732c3ba1 2023-04-05 18:54:23,235 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/297022b677ce4bc58adc5f92732c3ba1, entries=50, sequenceid=197, filesize=7.1 K 2023-04-05 18:54:23,236 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/26d8906213d648ec87c3fd0bcfd5148c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/26d8906213d648ec87c3fd0bcfd5148c 2023-04-05 18:54:23,240 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/26d8906213d648ec87c3fd0bcfd5148c, entries=50, sequenceid=197, filesize=7.1 K 2023-04-05 18:54:23,240 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=187.85 KB/192360 for 5f2134cd982ecee57a32108ad09a562d in 1702ms, sequenceid=197, compaction requested=true 2023-04-05 18:54:23,241 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:23,241 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:23,242 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:23,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:23,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:23,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:23,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:23,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:23,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eea0d26[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:23,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:23,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:23,250 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:23,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:23,250 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,250 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:23,252 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,252 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:23,252 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:23,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:23,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:23,252 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:23,257 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:23,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:23,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:23,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:23,257 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:23,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:23,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:23,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:23,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:23,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:23,260 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:23,263 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:23,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:23,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:23,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:23,266 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:23,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:23,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:23,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:23,267 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:23,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:23,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:23,267 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:23,267 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:23,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:23,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:23,367 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:23,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:23,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:23,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:23,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:23,371 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:23,371 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:23,371 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:23,371 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:23,372 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:23,372 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:23,373 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:23,373 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:23,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:23,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:23,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:23,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:23,376 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:23,376 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:23,376 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:23,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:23,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:23,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:23,379 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:23,380 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:23,380 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=187.85 KB heapSize=488.11 KB 2023-04-05 18:54:23,388 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:23,388 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:23,388 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:23,388 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:23,388 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:23,388 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:23,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:23,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:23,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:23,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720923752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:23,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720923753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:23,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720923760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:23,806 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405cdd5f81c90294b559b44812ba9d7f844 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cdd5f81c90294b559b44812ba9d7f844 2023-04-05 18:54:23,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:23,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720923855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:24,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55836 deadline: 1680720924057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,214 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=230, memsize=62.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/4b3d4f0446d34b51b4bce91a1b283322 2023-04-05 18:54:24,249 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/e4145cbf638646c88abf4818db25b9e0 2023-04-05 18:54:24,266 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/1d1e6040c5f847efb31fc05b977fbd46 2023-04-05 18:54:24,271 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/4b3d4f0446d34b51b4bce91a1b283322 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/4b3d4f0446d34b51b4bce91a1b283322 2023-04-05 18:54:24,275 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/4b3d4f0446d34b51b4bce91a1b283322, entries=150, sequenceid=230, filesize=25.3 K 2023-04-05 18:54:24,277 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/e4145cbf638646c88abf4818db25b9e0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e4145cbf638646c88abf4818db25b9e0 2023-04-05 18:54:24,283 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e4145cbf638646c88abf4818db25b9e0, entries=150, sequenceid=230, filesize=11.8 K 2023-04-05 18:54:24,284 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/1d1e6040c5f847efb31fc05b977fbd46 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1d1e6040c5f847efb31fc05b977fbd46 2023-04-05 18:54:24,290 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1d1e6040c5f847efb31fc05b977fbd46, entries=150, sequenceid=230, filesize=11.8 K 2023-04-05 18:54:24,291 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~187.85 KB/192360, heapSize ~488.06 KB/499776, currentSize=13.42 KB/13740 for 5f2134cd982ecee57a32108ad09a562d in 911ms, sequenceid=230, compaction requested=true 2023-04-05 18:54:24,291 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:24,291 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:24,291 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,297 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:24,297 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:24,299 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5043d3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:24,301 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:24,301 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:24,302 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:24,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:24,303 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,303 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:24,304 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:24,307 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:24,307 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:24,307 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:24,307 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:24,307 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:24,311 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:24,311 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:24,311 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:24,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:24,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:24,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:24,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:24,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:24,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:24,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:24,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:24,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:24,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:24,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:24,316 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:24,316 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:24,316 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:24,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:24,316 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:24,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:24,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:24,416 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:24,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:24,433 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:24,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:24,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:24,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:24,434 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:24,434 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:24,435 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:24,435 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:24,435 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,435 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:24,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:24,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:24,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:24,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:24,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:24,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:24,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:24,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:24,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:24,437 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:24,437 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:24,437 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 18:54:24,438 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:24,438 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:24,438 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:24,439 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:24,439 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:24,439 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:24,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:24,445 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:24,470 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040542bfb2936eae45f982c92e635fa3b0be to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040542bfb2936eae45f982c92e635fa3b0be 2023-04-05 18:54:24,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720864507, max timestamp=1680720864529Num uniques 859; 2023-04-05 18:54:24,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:24,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720864507, max timestamp=1680720864529Num uniques 859; 2023-04-05 18:54:24,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:24,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720864507, max timestamp=1680720864529Num uniques 859; 2023-04-05 18:54:24,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:24,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720864555, max timestamp=1680720864587Num uniques 1018; 2023-04-05 18:54:24,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720864555, max timestamp=1680720864587Num uniques 1018; 2023-04-05 18:54:24,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 18:54:24,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 18:54:24,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720864555, max timestamp=1680720864587Num uniques 1018; 2023-04-05 18:54:24,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 18:54:24,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720864644, max timestamp=1680720864660Num uniques 1027; 2023-04-05 18:54:24,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-05 18:54:24,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720864644, max timestamp=1680720864660Num uniques 1027; 2023-04-05 18:54:24,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-05 18:54:24,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720864644, max timestamp=1680720864660Num uniques 1027; 2023-04-05 18:54:24,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-05 18:54:24,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:24,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720924716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:24,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720924820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:24,918 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/a0828ca531274532bf399055ee4902e1 2023-04-05 18:54:25,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:25,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720925025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:25,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720925333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,358 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/5fc1378f5cee4c05a6126537f63b3e1e 2023-04-05 18:54:25,770 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/e6cc5486b4ef42bb905a2294fff922c0 2023-04-05 18:54:25,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:25,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720925768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,774 DEBUG [Thread-879] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:54:25,776 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/a0828ca531274532bf399055ee4902e1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a0828ca531274532bf399055ee4902e1 2023-04-05 18:54:25,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:25,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720925780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,780 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a0828ca531274532bf399055ee4902e1, entries=200, sequenceid=239, filesize=32.1 K 2023-04-05 18:54:25,782 DEBUG [Thread-887] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:54:25,782 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/5fc1378f5cee4c05a6126537f63b3e1e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/5fc1378f5cee4c05a6126537f63b3e1e 2023-04-05 18:54:25,787 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/5fc1378f5cee4c05a6126537f63b3e1e, entries=100, sequenceid=239, filesize=9.4 K 2023-04-05 18:54:25,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/e6cc5486b4ef42bb905a2294fff922c0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e6cc5486b4ef42bb905a2294fff922c0 2023-04-05 18:54:25,794 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e6cc5486b4ef42bb905a2294fff922c0, entries=100, sequenceid=239, filesize=9.4 K 2023-04-05 18:54:25,795 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 5f2134cd982ecee57a32108ad09a562d in 1358ms, sequenceid=239, compaction requested=true 2023-04-05 18:54:25,795 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:25,795 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:25,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:25,805 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,805 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:25,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:25,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b7b5f7f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:25,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:25,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,814 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:25,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:25,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:25,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:25,816 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:25,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:25,818 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:25,818 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:25,818 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:25,818 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:25,818 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:25,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:25,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:25,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:25,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:25,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:25,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:25,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:25,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:25,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:25,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:25,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:25,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,826 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:25,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,826 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:25,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:25,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:25,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=147.60 KB heapSize=419.16 KB 2023-04-05 18:54:25,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:25,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:25,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:25,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:25,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:25,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:25,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:25,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720925859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,926 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:25,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:25,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:25,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:25,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:25,930 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:25,930 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:25,930 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:25,930 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:25,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:25,932 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:25,932 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:25,933 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,933 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:25,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:25,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:25,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:25,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:25,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:25,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:25,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:25,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:25,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,936 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:25,937 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:25,937 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:25,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:25,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6f94cb[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:25,938 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:25,938 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,940 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:25,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:25,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:25,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:25,942 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,942 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:25,942 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:25,942 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:25,944 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:25,944 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:25,944 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:25,944 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:25,944 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:25,946 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,946 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,946 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:25,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:25,946 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:25,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:25,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:25,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:25,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:25,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:25,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:25,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:25,951 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:25,951 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,951 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:25,951 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:25,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:25,951 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:25,951 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:25,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:25,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720925963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,051 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:26,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:26,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:26,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:26,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:26,055 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:26,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:26,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:26,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,057 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:26,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:26,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:26,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:26,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:26,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:26,061 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:26,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1112273d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:26,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:26,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,064 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:26,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:26,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:26,064 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,064 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:26,066 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:26,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:26,069 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:26,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:26,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:26,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:26,069 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:26,072 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,072 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,073 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,073 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:26,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:26,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:26,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,077 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:26,077 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:26,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:26,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720926168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,178 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:26,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:26,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:26,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:26,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:26,181 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:26,181 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,181 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:26,181 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:26,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,183 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,183 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:26,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:26,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:26,185 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,187 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:26,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ca1a017[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:26,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:26,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:26,196 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,196 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:26,197 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:26,200 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:26,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:26,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:26,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:26,200 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:26,202 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,202 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,202 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,204 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:26,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:26,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,208 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:26,208 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:26,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,256 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405ed7475dbdfed4fc28ce55d4bf813d711 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ed7475dbdfed4fc28ce55d4bf813d711 2023-04-05 18:54:26,261 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/21c3002db287487aab1c8a38a2d09e7e 2023-04-05 18:54:26,308 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:26,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:26,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:26,311 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:26,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:26,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:26,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:26,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:26,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:26,315 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,315 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:26,315 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:26,315 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:26,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:26,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:26,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d208a5b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:26,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,320 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:26,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:26,320 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,320 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:26,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:26,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:26,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:26,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:26,325 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:26,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:26,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:26,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:26,325 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:26,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,327 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:26,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:26,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,331 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:26,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,332 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:26,332 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:26,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,432 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:26,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:26,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:26,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:26,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:26,436 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:26,436 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,436 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:26,436 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:26,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,438 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:26,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:26,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:26,440 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,440 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:26,440 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:26,440 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:26,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:26,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:26,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7efa82f8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:26,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:26,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:26,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:26,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:26,447 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,447 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:26,449 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:26,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:26,451 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:26,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:26,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:26,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:26,451 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:26,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,455 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,462 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:26,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:26,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:26,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,463 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:26,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,463 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:26,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720926472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,563 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:26,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:26,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:26,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:26,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:26,566 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:26,567 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,567 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:26,567 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:26,567 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,567 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,568 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,569 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,569 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:26,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:26,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:26,570 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,570 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:26,570 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:26,570 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,571 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:26,571 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:26,572 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:26,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed169d5[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:26,574 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:26,574 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:26,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:26,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:26,576 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,576 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:26,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:26,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:26,580 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:26,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:26,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:26,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:26,581 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:26,584 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,584 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,589 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,589 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,589 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,589 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,589 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:26,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:26,589 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:26,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,590 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:26,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,590 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:26,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55752 deadline: 1680720926656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,662 DEBUG [Thread-885] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:54:26,670 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/a1f76fd11b1f449f95cd278ea0652ef8 2023-04-05 18:54:26,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:26,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55820 deadline: 1680720926678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,687 DEBUG [Thread-883] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:54:26,690 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:26,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:26,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:26,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:26,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:26,693 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:26,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:26,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:26,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,695 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,695 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,695 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,695 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:26,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:26,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:26,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:26,697 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,697 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:26,697 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:26,697 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:26,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f11df00[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:26,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:26,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,702 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:26,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:26,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:26,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:26,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:26,706 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:26,706 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:26,706 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:26,706 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:26,706 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:26,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:26,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:26,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:26,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,715 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:26,716 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,816 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:26,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:26,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:26,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:26,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:26,821 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:26,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:26,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:26,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,823 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:26,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,824 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:26,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:26,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:26,828 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,828 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:26,828 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:26,828 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:26,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:26,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:26,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:26,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,830 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:26,832 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:26,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@760afceb[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:26,832 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,833 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:26,833 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:26,833 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:26,833 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,833 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:26,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:26,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:26,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:26,838 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:26,838 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:26,838 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:26,838 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:26,838 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:26,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,840 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,841 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,843 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,843 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:26,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:26,843 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,843 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,843 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,844 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:26,844 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:26,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:26,944 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:26,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:26,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:26,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:26,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:26,947 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:26,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:26,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:26,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,949 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:26,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:26,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:26,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:26,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:26,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:26,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4d5d90[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:26,959 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:26,959 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,961 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,961 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:26,961 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:26,961 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:26,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:26,962 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:26,962 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,962 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,962 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:26,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:26,969 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:26,969 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:26,969 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:26,969 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:26,969 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:26,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:26,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:26,971 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,974 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:26,980 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:26,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:26,980 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:26,980 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:26,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:26,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:26,980 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:26,980 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:26,980 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,980 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:26,980 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:26,980 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:26,980 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:26,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:26,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:26,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720926980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,080 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:27,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:27,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:27,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:27,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:27,084 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:27,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:27,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:27,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:27,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:27,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,086 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:27,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:27,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:27,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:27,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:27,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:27,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:27,087 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:27,087 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:27,087 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:27,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:27,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:27,088 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:27,088 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:27,088 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:27,088 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:27,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:27,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:27,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:27,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:27,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,090 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:27,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:27,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/df64313ae19e4b46a626621f93a9e846 2023-04-05 18:54:27,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:27,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:27,091 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:27,091 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:27,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:27,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2927180d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:27,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:27,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:27,094 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:27,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:27,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:27,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:27,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:27,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:27,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:27,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/21c3002db287487aab1c8a38a2d09e7e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/21c3002db287487aab1c8a38a2d09e7e 2023-04-05 18:54:27,096 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:27,096 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:27,096 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,096 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:27,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/21c3002db287487aab1c8a38a2d09e7e, entries=200, sequenceid=311, filesize=32.3 K 2023-04-05 18:54:27,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:27,100 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:27,100 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:27,100 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:27,100 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:27,100 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:27,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/a1f76fd11b1f449f95cd278ea0652ef8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a1f76fd11b1f449f95cd278ea0652ef8 2023-04-05 18:54:27,102 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,102 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:27,102 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:27,104 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,105 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a1f76fd11b1f449f95cd278ea0652ef8, entries=150, sequenceid=311, filesize=11.9 K 2023-04-05 18:54:27,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:27,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:27,108 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:27,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:27,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:27,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:27,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:27,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:27,109 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:27,109 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,109 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:27,109 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:27,109 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,109 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:27,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:27,109 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:27,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/df64313ae19e4b46a626621f93a9e846 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/df64313ae19e4b46a626621f93a9e846 2023-04-05 18:54:27,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:27,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/df64313ae19e4b46a626621f93a9e846, entries=150, sequenceid=311, filesize=11.9 K 2023-04-05 18:54:27,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~419.11 KB/429168, currentSize=33.54 KB/34350 for 5f2134cd982ecee57a32108ad09a562d in 1274ms, sequenceid=311, compaction requested=true 2023-04-05 18:54:27,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:27,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:27,114 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 18:54:27,114 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 18:54:27,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:27,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:27,117 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 188538 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 18:54:27,117 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/A is initiating minor compaction (all files) 2023-04-05 18:54:27,117 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/A in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:27,117 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5ebcfbb56cba4d60a7e42f327f6aafc7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/fd79a13abcce4fef97405bfe9406fd02, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/e5a944ac0fcd4bab868c4ee0c26e53fb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5604ba7064494a83a4685f3a766a1a58, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/4b3d4f0446d34b51b4bce91a1b283322, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a0828ca531274532bf399055ee4902e1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/21c3002db287487aab1c8a38a2d09e7e] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=184.1 K 2023-04-05 18:54:27,118 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 5ebcfbb56cba4d60a7e42f327f6aafc7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680720858170 2023-04-05 18:54:27,118 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 77312 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 18:54:27,118 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/B is initiating minor compaction (all files) 2023-04-05 18:54:27,118 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/B in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:27,118 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/674bbcdd9b9c4a9486350b962d90870d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/77930071eefe45ae99af88b3a571b37a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/23363e235bcf4fbcaf1fea63d1838ff6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/297022b677ce4bc58adc5f92732c3ba1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e4145cbf638646c88abf4818db25b9e0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/5fc1378f5cee4c05a6126537f63b3e1e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a1f76fd11b1f449f95cd278ea0652ef8] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=75.5 K 2023-04-05 18:54:27,118 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting fd79a13abcce4fef97405bfe9406fd02, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680720858486 2023-04-05 18:54:27,119 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 674bbcdd9b9c4a9486350b962d90870d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680720858170 2023-04-05 18:54:27,119 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting e5a944ac0fcd4bab868c4ee0c26e53fb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680720860335 2023-04-05 18:54:27,119 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 77930071eefe45ae99af88b3a571b37a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680720858486 2023-04-05 18:54:27,119 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 5604ba7064494a83a4685f3a766a1a58, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680720861457 2023-04-05 18:54:27,120 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 23363e235bcf4fbcaf1fea63d1838ff6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680720860335 2023-04-05 18:54:27,120 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 4b3d4f0446d34b51b4bce91a1b283322, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680720861618 2023-04-05 18:54:27,121 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting a0828ca531274532bf399055ee4902e1, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680720864388 2023-04-05 18:54:27,121 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 297022b677ce4bc58adc5f92732c3ba1, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680720861459 2023-04-05 18:54:27,122 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 21c3002db287487aab1c8a38a2d09e7e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680720864709 2023-04-05 18:54:27,122 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e4145cbf638646c88abf4818db25b9e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680720861618 2023-04-05 18:54:27,123 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 5fc1378f5cee4c05a6126537f63b3e1e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680720864405 2023-04-05 18:54:27,123 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a1f76fd11b1f449f95cd278ea0652ef8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680720864709 2023-04-05 18:54:27,147 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#B#compaction#156 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:27,156 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#A#compaction#155 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:27,172 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/31cc92b239e848d5bdb1f004b9ef9ed6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/31cc92b239e848d5bdb1f004b9ef9ed6 2023-04-05 18:54:27,177 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/B of 5f2134cd982ecee57a32108ad09a562d into 31cc92b239e848d5bdb1f004b9ef9ed6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:27,177 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:27,177 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/B, priority=9, startTime=1680720867114; duration=0sec 2023-04-05 18:54:27,178 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:27,178 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 18:54:27,181 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 77312 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 18:54:27,181 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/C is initiating minor compaction (all files) 2023-04-05 18:54:27,181 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/C in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:27,181 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a614455a1b6a4425ae4f0e0d51eaf039, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a80ac4372444416d817067f6c350d45d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/893e22f04f734714a34af5ca89654a9c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/26d8906213d648ec87c3fd0bcfd5148c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1d1e6040c5f847efb31fc05b977fbd46, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e6cc5486b4ef42bb905a2294fff922c0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/df64313ae19e4b46a626621f93a9e846] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=75.5 K 2023-04-05 18:54:27,182 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a614455a1b6a4425ae4f0e0d51eaf039, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680720858170 2023-04-05 18:54:27,182 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a80ac4372444416d817067f6c350d45d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680720858486 2023-04-05 18:54:27,182 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 893e22f04f734714a34af5ca89654a9c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680720860335 2023-04-05 18:54:27,183 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 26d8906213d648ec87c3fd0bcfd5148c, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680720861459 2023-04-05 18:54:27,183 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1d1e6040c5f847efb31fc05b977fbd46, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680720861618 2023-04-05 18:54:27,184 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e6cc5486b4ef42bb905a2294fff922c0, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680720864405 2023-04-05 18:54:27,184 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting df64313ae19e4b46a626621f93a9e846, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680720864709 2023-04-05 18:54:27,209 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:27,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:27,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:27,224 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#C#compaction#157 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:27,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:27,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:27,235 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:27,235 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:27,235 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:27,235 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:27,235 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,235 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:27,237 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,237 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:27,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:27,238 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:27,238 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:27,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:27,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:27,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:27,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:27,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:27,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:27,252 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:27,252 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:27,252 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:27,252 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:27,252 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 18:54:27,252 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:27,252 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:27,252 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:27,253 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:27,253 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:27,253 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:27,652 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/a72b92a6528a43b5986dd7cc5a2443e7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a72b92a6528a43b5986dd7cc5a2443e7 2023-04-05 18:54:27,655 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/ca1babfdc7b74b488b4597c30b576f57 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/ca1babfdc7b74b488b4597c30b576f57 2023-04-05 18:54:27,658 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/A of 5f2134cd982ecee57a32108ad09a562d into a72b92a6528a43b5986dd7cc5a2443e7(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:27,658 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:27,658 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/A, priority=9, startTime=1680720867114; duration=0sec 2023-04-05 18:54:27,658 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:27,660 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/C of 5f2134cd982ecee57a32108ad09a562d into ca1babfdc7b74b488b4597c30b576f57(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:27,660 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:27,660 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/C, priority=9, startTime=1680720867115; duration=0sec 2023-04-05 18:54:27,660 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:27,667 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405826e9c778a554e849af4f873405a7b51 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405826e9c778a554e849af4f873405a7b51 2023-04-05 18:54:27,673 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=320, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/a2310e3324c04f188c7fe2efd7764258 2023-04-05 18:54:27,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:27,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:28,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720868014, max timestamp=1680720868028Num uniques 859; 2023-04-05 18:54:28,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:28,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720868014, max timestamp=1680720868028Num uniques 859; 2023-04-05 18:54:28,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720868014, max timestamp=1680720868028Num uniques 859; 2023-04-05 18:54:28,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:28,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:28,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720868056, max timestamp=1680720868065Num uniques 1068; 2023-04-05 18:54:28,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-05 18:54:28,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720868056, max timestamp=1680720868065Num uniques 1068; 2023-04-05 18:54:28,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720868056, max timestamp=1680720868065Num uniques 1068; 2023-04-05 18:54:28,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-05 18:54:28,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-05 18:54:28,085 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/73847a894a3342e781210bdc532b9959 2023-04-05 18:54:28,098 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/4ffde19f034e43aca6970b7fe2219b8b 2023-04-05 18:54:28,102 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/a2310e3324c04f188c7fe2efd7764258 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a2310e3324c04f188c7fe2efd7764258 2023-04-05 18:54:28,107 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a2310e3324c04f188c7fe2efd7764258, entries=150, sequenceid=320, filesize=25.4 K 2023-04-05 18:54:28,107 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/73847a894a3342e781210bdc532b9959 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/73847a894a3342e781210bdc532b9959 2023-04-05 18:54:28,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720868097, max timestamp=1680720868114Num uniques 1027; 2023-04-05 18:54:28,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720868097, max timestamp=1680720868114Num uniques 1027; 2023-04-05 18:54:28,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720868097, max timestamp=1680720868114Num uniques 1027; 2023-04-05 18:54:28,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-05 18:54:28,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-05 18:54:28,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-05 18:54:28,118 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/73847a894a3342e781210bdc532b9959, entries=150, sequenceid=320, filesize=11.9 K 2023-04-05 18:54:28,120 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/4ffde19f034e43aca6970b7fe2219b8b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/4ffde19f034e43aca6970b7fe2219b8b 2023-04-05 18:54:28,124 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/4ffde19f034e43aca6970b7fe2219b8b, entries=150, sequenceid=320, filesize=11.9 K 2023-04-05 18:54:28,125 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=53.67 KB/54960 for 5f2134cd982ecee57a32108ad09a562d in 873ms, sequenceid=320, compaction requested=false 2023-04-05 18:54:28,125 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:28,125 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:28,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:28,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:28,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:28,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:28,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,126 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:28,127 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,127 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:28,127 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=60.38 KB heapSize=159.47 KB 2023-04-05 18:54:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:28,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:28,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:28,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:28,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385f2bf6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:28,129 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:28,129 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,130 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,131 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:28,131 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:28,131 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:28,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:28,132 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:28,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:28,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:28,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:28,135 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:28,135 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:28,135 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:28,135 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:28,135 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:28,138 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,138 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:28,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,139 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:28,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,140 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,142 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:28,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,145 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:28,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:28,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:28,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:28,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:28,145 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:28,145 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:28,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:28,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,146 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,146 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,146 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,146 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:28,171 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405395f7e9d68894ce1bc19a2e9780c5847 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405395f7e9d68894ce1bc19a2e9780c5847 2023-04-05 18:54:28,201 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=381, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/1ceb03ee2a0340c89609c595868f3343 2023-04-05 18:54:28,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720868187, max timestamp=1680720868213Num uniques 859; 2023-04-05 18:54:28,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:28,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720868187, max timestamp=1680720868213Num uniques 859; 2023-04-05 18:54:28,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720868187, max timestamp=1680720868213Num uniques 859; 2023-04-05 18:54:28,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:28,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:28,218 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/7e7424971e534d249b072fc6a470f839 2023-04-05 18:54:28,245 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:28,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:28,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:28,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:28,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:28,249 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:28,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:28,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:28,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:28,250 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,250 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,251 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:28,251 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:28,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:28,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:28,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:28,254 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:28,254 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:28,254 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:28,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:28,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:28,255 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:28,255 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:28,255 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:28,255 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:28,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:28,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:28,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:28,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:28,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:28,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,257 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,258 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,258 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:28,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@199375b2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:28,259 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:28,259 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,261 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:28,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:28,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:28,261 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,261 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:28,262 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,262 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,262 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:28,262 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:28,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:28,265 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:28,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:28,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:28,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:28,265 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:28,267 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,267 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,267 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,269 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:28,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720928269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:28,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:28,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:28,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,274 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:28,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,275 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:28,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:28,275 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:28,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:28,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:28,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:28,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:28,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720928373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,375 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:28,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:28,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:28,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:28,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:28,378 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:28,378 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:28,378 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:28,378 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:28,379 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,379 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:28,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:28,380 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,380 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,381 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:28,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:28,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:28,381 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:28,381 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:28,381 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:28,381 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:28,382 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:28,382 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:28,382 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:28,382 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:28,384 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:28,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f841acd[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:28,386 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:28,386 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,387 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:28,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:28,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:28,390 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:28,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:28,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:28,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:28,394 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,394 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:28,394 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:28,394 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:28,394 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:28,394 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:28,394 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:28,395 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,395 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,396 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,418 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,419 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,421 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,421 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:28,421 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:28,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:28,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:28,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,422 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:28,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:28,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,422 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:28,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,422 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:28,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:28,522 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:28,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:28,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:28,525 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:28,525 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:28,525 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:28,525 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:28,526 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,526 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,527 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,527 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:28,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:28,527 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,528 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:28,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:28,529 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:28,529 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:28,529 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:28,529 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:28,531 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,531 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:28,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e728840[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:28,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:28,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,535 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:28,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:28,536 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:28,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:28,539 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:28,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:28,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:28,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:28,539 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:28,541 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,541 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:28,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,541 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:28,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:28,542 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,543 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,549 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:28,549 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:28,549 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:28,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:28,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:28,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:28,549 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:28,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,550 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:28,550 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:28,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:28,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:28,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720928575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/59c058345f794e7e8eda2bbf37751018 2023-04-05 18:54:28,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/1ceb03ee2a0340c89609c595868f3343 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/1ceb03ee2a0340c89609c595868f3343 2023-04-05 18:54:28,641 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/1ceb03ee2a0340c89609c595868f3343, entries=200, sequenceid=381, filesize=32.3 K 2023-04-05 18:54:28,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/7e7424971e534d249b072fc6a470f839 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7e7424971e534d249b072fc6a470f839 2023-04-05 18:54:28,650 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:28,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7e7424971e534d249b072fc6a470f839, entries=150, sequenceid=381, filesize=11.9 K 2023-04-05 18:54:28,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:28,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/59c058345f794e7e8eda2bbf37751018 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/59c058345f794e7e8eda2bbf37751018 2023-04-05 18:54:28,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:28,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:28,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:28,655 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:28,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:28,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:28,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:28,656 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,656 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/59c058345f794e7e8eda2bbf37751018, entries=150, sequenceid=381, filesize=11.9 K 2023-04-05 18:54:28,657 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:28,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:28,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:28,658 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:28,658 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:28,658 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~159.42 KB/163248, currentSize=127.47 KB/130530 for 5f2134cd982ecee57a32108ad09a562d in 531ms, sequenceid=381, compaction requested=true 2023-04-05 18:54:28,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:28,658 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:28,658 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:54:28,658 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-05 18:54:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,658 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:28,659 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:28,660 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:28,660 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85656 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:28,660 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/C is initiating minor compaction (all files) 2023-04-05 18:54:28,660 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/A is initiating minor compaction (all files) 2023-04-05 18:54:28,660 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/C in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:28,660 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/A in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:28,660 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/ca1babfdc7b74b488b4597c30b576f57, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/4ffde19f034e43aca6970b7fe2219b8b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/59c058345f794e7e8eda2bbf37751018] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=36.2 K 2023-04-05 18:54:28,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:28,660 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a72b92a6528a43b5986dd7cc5a2443e7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a2310e3324c04f188c7fe2efd7764258, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/1ceb03ee2a0340c89609c595868f3343] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=83.6 K 2023-04-05 18:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:28,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:28,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:28,661 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:28,661 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:28,661 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:28,661 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting ca1babfdc7b74b488b4597c30b576f57, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680720864709 2023-04-05 18:54:28,662 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a72b92a6528a43b5986dd7cc5a2443e7, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680720864709 2023-04-05 18:54:28,662 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:28,662 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 4ffde19f034e43aca6970b7fe2219b8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680720865850 2023-04-05 18:54:28,662 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:28,662 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:28,662 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a2310e3324c04f188c7fe2efd7764258, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680720865850 2023-04-05 18:54:28,662 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:28,662 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-05 18:54:28,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:28,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:28,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:28,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:28,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:28,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:28,664 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 59c058345f794e7e8eda2bbf37751018, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680720868119 2023-04-05 18:54:28,664 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1ceb03ee2a0340c89609c595868f3343, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680720868111 2023-04-05 18:54:28,673 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#C#compaction#166 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:28,694 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#A#compaction#165 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:28,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:28,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:28,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:28,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720928907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:29,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:29,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720929012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:29,106 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/179c1350edfa49fa9b2e27ec1033f2cf as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/179c1350edfa49fa9b2e27ec1033f2cf 2023-04-05 18:54:29,112 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304050f11ebcf58a148e396e63391a630b577 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050f11ebcf58a148e396e63391a630b577 2023-04-05 18:54:29,116 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/C of 5f2134cd982ecee57a32108ad09a562d into 179c1350edfa49fa9b2e27ec1033f2cf(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:29,116 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:29,116 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/C, priority=13, startTime=1680720868658; duration=0sec 2023-04-05 18:54:29,116 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:29,118 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:29,119 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:29,119 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/B is initiating minor compaction (all files) 2023-04-05 18:54:29,119 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/B in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:29,119 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/31cc92b239e848d5bdb1f004b9ef9ed6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/73847a894a3342e781210bdc532b9959, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7e7424971e534d249b072fc6a470f839] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=36.2 K 2023-04-05 18:54:29,120 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 31cc92b239e848d5bdb1f004b9ef9ed6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680720864709 2023-04-05 18:54:29,121 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 73847a894a3342e781210bdc532b9959, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680720865850 2023-04-05 18:54:29,121 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7e7424971e534d249b072fc6a470f839, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680720868119 2023-04-05 18:54:29,133 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#B#compaction#167 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:29,164 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/3610432751204ceda09e0b6865b4da65 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/3610432751204ceda09e0b6865b4da65 2023-04-05 18:54:29,169 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/A of 5f2134cd982ecee57a32108ad09a562d into 3610432751204ceda09e0b6865b4da65(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:29,169 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:29,169 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/A, priority=13, startTime=1680720868658; duration=0sec 2023-04-05 18:54:29,172 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:29,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:29,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720929216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:29,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:29,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720929521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:29,558 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=416, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/6eac8ad2269d4ed7b774681b667fc092 2023-04-05 18:54:29,567 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/ea4efeebaa684ff7bba00ff637389e63 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ea4efeebaa684ff7bba00ff637389e63 2023-04-05 18:54:29,578 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/B of 5f2134cd982ecee57a32108ad09a562d into ea4efeebaa684ff7bba00ff637389e63(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:29,578 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:29,578 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/B, priority=13, startTime=1680720868658; duration=0sec 2023-04-05 18:54:29,578 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:29,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:29,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55790 deadline: 1680720929799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:29,803 DEBUG [Thread-887] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:54:29,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:29,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55806 deadline: 1680720929804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:29,807 DEBUG [Thread-879] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:54:29,974 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/1fa5a804ed9c44e193d8683e35b7bd73 2023-04-05 18:54:30,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:30,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720930025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,388 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/74ddfc16796842d5bac1fe1dcae1927c 2023-04-05 18:54:30,393 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/6eac8ad2269d4ed7b774681b667fc092 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6eac8ad2269d4ed7b774681b667fc092 2023-04-05 18:54:30,397 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6eac8ad2269d4ed7b774681b667fc092, entries=150, sequenceid=416, filesize=25.4 K 2023-04-05 18:54:30,398 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/1fa5a804ed9c44e193d8683e35b7bd73 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1fa5a804ed9c44e193d8683e35b7bd73 2023-04-05 18:54:30,403 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1fa5a804ed9c44e193d8683e35b7bd73, entries=150, sequenceid=416, filesize=11.9 K 2023-04-05 18:54:30,404 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/74ddfc16796842d5bac1fe1dcae1927c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/74ddfc16796842d5bac1fe1dcae1927c 2023-04-05 18:54:30,414 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/74ddfc16796842d5bac1fe1dcae1927c, entries=150, sequenceid=416, filesize=11.9 K 2023-04-05 18:54:30,415 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 5f2134cd982ecee57a32108ad09a562d in 1753ms, sequenceid=416, compaction requested=false 2023-04-05 18:54:30,415 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:30,415 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:30,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,418 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,418 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3314e3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:30,419 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:30,419 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,421 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:30,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:30,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:30,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:30,425 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:30,426 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:30,426 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:30,426 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:30,426 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:30,431 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:30,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,434 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,436 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,436 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:30,436 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:30,436 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:30,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:30,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:30,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:30,437 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:30,437 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:30,436 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,436 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:30,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:30,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:30,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,537 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:30,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:30,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:30,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:30,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:30,542 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:30,542 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:30,542 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:30,542 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:30,542 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,543 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,544 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:30,544 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,544 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,544 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:30,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:30,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:30,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:30,546 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:30,546 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:30,580 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040552df13d78e8e49efb235181af45a6f61 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040552df13d78e8e49efb235181af45a6f61 2023-04-05 18:54:30,592 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=430, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/361d5f3d486f425790629897044fc287 2023-04-05 18:54:30,606 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/a89d095256ca4aca90f23b7c6d742fce 2023-04-05 18:54:30,617 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/cc714a157b6d404fb99669e27c04d937 2023-04-05 18:54:30,621 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/361d5f3d486f425790629897044fc287 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/361d5f3d486f425790629897044fc287 2023-04-05 18:54:30,626 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/361d5f3d486f425790629897044fc287, entries=150, sequenceid=430, filesize=25.4 K 2023-04-05 18:54:30,628 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/a89d095256ca4aca90f23b7c6d742fce as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a89d095256ca4aca90f23b7c6d742fce 2023-04-05 18:54:30,632 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a89d095256ca4aca90f23b7c6d742fce, entries=150, sequenceid=430, filesize=11.9 K 2023-04-05 18:54:30,633 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/cc714a157b6d404fb99669e27c04d937 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/cc714a157b6d404fb99669e27c04d937 2023-04-05 18:54:30,642 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/cc714a157b6d404fb99669e27c04d937, entries=150, sequenceid=430, filesize=11.9 K 2023-04-05 18:54:30,643 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 5f2134cd982ecee57a32108ad09a562d in 97ms, sequenceid=430, compaction requested=true 2023-04-05 18:54:30,643 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:30,643 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:30,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,653 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,653 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:30,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:30,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d0fc052[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:30,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:30,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:30,657 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:30,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:30,659 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,659 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,659 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:30,659 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:30,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:30,662 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:30,662 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:30,662 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:30,662 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:30,663 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:30,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:30,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,667 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,673 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,673 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,673 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,673 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,673 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,673 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,673 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:30,673 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:30,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:30,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:30,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:30,673 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:30,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:30,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:30,674 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:30,674 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:30,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:30,774 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:30,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:30,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:30,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:30,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:30,778 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:30,778 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:30,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:30,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:30,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,781 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:30,781 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:30,781 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,781 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:30,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:30,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:30,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:30,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:30,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:30,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:30,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:30,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:30,784 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:30,785 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:30,785 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:30,785 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:30,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:30,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:30,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:30,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:30,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7252ef4c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:30,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:30,790 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,791 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,791 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:30,792 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:30,792 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:30,792 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,792 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:30,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:30,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:30,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:30,794 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:30,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:30,797 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:30,797 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:30,797 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:30,797 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:30,797 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:30,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:30,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,800 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,803 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,805 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:30,805 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:30,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:30,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:30,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:30,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:30,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,806 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:30,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:30,806 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:30,806 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:30,906 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:30,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:30,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:30,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:30,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:30,913 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:30,913 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:30,913 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:30,913 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:30,913 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,913 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,915 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:30,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:30,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:30,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:30,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:30,917 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:30,917 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:30,917 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:30,917 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:30,917 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:30,920 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:30,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58c859ce[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:30,922 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:30,922 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,926 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:30,926 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:30,926 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:30,926 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,926 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:30,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:30,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:30,928 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:30,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:30,931 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:30,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:30,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:30,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:30,931 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:30,933 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,933 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:30,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,933 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:30,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:30,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:30,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,934 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:30,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,939 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:30,939 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:30,939 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:30,939 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:30,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:30,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:30,939 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:30,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:30,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:30,940 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:30,940 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:30,940 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,940 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:30,940 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:30,940 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:30,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:30,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:31,040 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:31,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:31,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:31,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:31,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:31,045 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:31,045 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:31,045 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:31,045 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:31,045 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:31,045 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:31,046 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:31,047 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:31,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:31,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:31,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:31,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:31,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:31,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:31,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:31,048 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:31,048 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:31,048 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:31,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:31,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:31,052 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:31,052 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:31,052 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-05 18:54:31,052 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:31,052 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:31,052 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:31,053 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:31,053 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:31,053 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:31,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:31,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:31,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720871125, max timestamp=1680720871138Num uniques 859; 2023-04-05 18:54:31,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720871125, max timestamp=1680720871138Num uniques 859; 2023-04-05 18:54:31,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:31,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720871125, max timestamp=1680720871138Num uniques 859; 2023-04-05 18:54:31,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:31,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:31,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680720871188, max timestamp=1680720871204Num uniques 1068; 2023-04-05 18:54:31,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680720871188, max timestamp=1680720871204Num uniques 1068; 2023-04-05 18:54:31,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-05 18:54:31,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-05 18:54:31,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680720871188, max timestamp=1680720871204Num uniques 1068; 2023-04-05 18:54:31,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-05 18:54:31,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720871242, max timestamp=1680720871255Num uniques 1127; 2023-04-05 18:54:31,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720871242, max timestamp=1680720871255Num uniques 1127; 2023-04-05 18:54:31,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720871242, max timestamp=1680720871255Num uniques 1127; 2023-04-05 18:54:31,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-05 18:54:31,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-05 18:54:31,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-05 18:54:31,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680720871293, max timestamp=1680720871326Num uniques 1086; 2023-04-05 18:54:31,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680720871293, max timestamp=1680720871326Num uniques 1086; 2023-04-05 18:54:31,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-05 18:54:31,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680720871293, max timestamp=1680720871326Num uniques 1086; 2023-04-05 18:54:31,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-05 18:54:31,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-05 18:54:31,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680720871417, max timestamp=1680720871449Num uniques 1095; 2023-04-05 18:54:31,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-05 18:54:31,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680720871417, max timestamp=1680720871449Num uniques 1095; 2023-04-05 18:54:31,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680720871417, max timestamp=1680720871449Num uniques 1095; 2023-04-05 18:54:31,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-05 18:54:31,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-05 18:54:31,494 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040536cbe510dedf482a920b4fd4165bd360 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040536cbe510dedf482a920b4fd4165bd360 2023-04-05 18:54:31,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720871497, max timestamp=1680720871531Num uniques 1054; 2023-04-05 18:54:31,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 18:54:31,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720871497, max timestamp=1680720871531Num uniques 1104; 2023-04-05 18:54:31,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720871497, max timestamp=1680720871531Num uniques 1104; 2023-04-05 18:54:31,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-05 18:54:31,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-05 18:54:31,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680720871564, max timestamp=1680720871642Num uniques 1013; 2023-04-05 18:54:31,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-05 18:54:31,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680720871564, max timestamp=1680720871642Num uniques 1013; 2023-04-05 18:54:31,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-05 18:54:31,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680720871564, max timestamp=1680720871642Num uniques 1013; 2023-04-05 18:54:31,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-05 18:54:31,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680720871688, max timestamp=1680720871715Num uniques 1072; 2023-04-05 18:54:31,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-05 18:54:31,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680720871688, max timestamp=1680720871715Num uniques 1072; 2023-04-05 18:54:31,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-05 18:54:31,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680720871688, max timestamp=1680720871715Num uniques 1022; 2023-04-05 18:54:31,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-05 18:54:31,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680720871747, max timestamp=1680720871798Num uniques 1031; 2023-04-05 18:54:31,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-05 18:54:31,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680720871747, max timestamp=1680720871798Num uniques 1031; 2023-04-05 18:54:31,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-05 18:54:31,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680720871747, max timestamp=1680720871798Num uniques 1031; 2023-04-05 18:54:31,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-05 18:54:31,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680720871831, max timestamp=1680720871866Num uniques 1040; 2023-04-05 18:54:31,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-05 18:54:31,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680720871831, max timestamp=1680720871866Num uniques 1040; 2023-04-05 18:54:31,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-05 18:54:31,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680720871831, max timestamp=1680720871866Num uniques 1040; 2023-04-05 18:54:31,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-05 18:54:31,908 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=436, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/7a5b684a6e984128a05b1207f4e30872 2023-04-05 18:54:31,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680720871907, max timestamp=1680720871950Num uniques 1049; 2023-04-05 18:54:31,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-05 18:54:31,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680720871907, max timestamp=1680720871950Num uniques 1049; 2023-04-05 18:54:31,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-05 18:54:31,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680720871907, max timestamp=1680720871950Num uniques 1049; 2023-04-05 18:54:31,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-05 18:54:32,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680720871981, max timestamp=1680720872007Num uniques 1058; 2023-04-05 18:54:32,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-05 18:54:32,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680720871981, max timestamp=1680720872007Num uniques 1058; 2023-04-05 18:54:32,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-05 18:54:32,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680720871981, max timestamp=1680720872007Num uniques 1058; 2023-04-05 18:54:32,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-05 18:54:32,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680720872034, max timestamp=1680720872057Num uniques 1067; 2023-04-05 18:54:32,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-05 18:54:32,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680720872034, max timestamp=1680720872057Num uniques 1067; 2023-04-05 18:54:32,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680720872034, max timestamp=1680720872057Num uniques 1067; 2023-04-05 18:54:32,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-05 18:54:32,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-05 18:54:32,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680720872095, max timestamp=1680720872103Num uniques 1126; 2023-04-05 18:54:32,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=8056, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-05 18:54:32,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680720872090, max timestamp=1680720872103Num uniques 1126; 2023-04-05 18:54:32,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-05 18:54:32,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680720872095, max timestamp=1680720872103Num uniques 1126; 2023-04-05 18:54:32,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=8056, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-05 18:54:32,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680720872129, max timestamp=1680720872163Num uniques 1035; 2023-04-05 18:54:32,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680720872129, max timestamp=1680720872163Num uniques 1085; 2023-04-05 18:54:32,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-05 18:54:32,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-05 18:54:32,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680720872129, max timestamp=1680720872163Num uniques 1035; 2023-04-05 18:54:32,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-05 18:54:32,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680720872204, max timestamp=1680720872212Num uniques 1044; 2023-04-05 18:54:32,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-05 18:54:32,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680720872204, max timestamp=1680720872212Num uniques 1044; 2023-04-05 18:54:32,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-05 18:54:32,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680720872204, max timestamp=1680720872212Num uniques 1044; 2023-04-05 18:54:32,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-05 18:54:32,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680720872248, max timestamp=1680720872273Num uniques 1053; 2023-04-05 18:54:32,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-05 18:54:32,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720872255, max timestamp=1680720872273Num uniques 1053; 2023-04-05 18:54:32,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720872255, max timestamp=1680720872273Num uniques 1053; 2023-04-05 18:54:32,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=6870, suffix heap size=137480,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=150 2023-04-05 18:54:32,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=6870, suffix heap size=137480,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=150 2023-04-05 18:54:32,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1680720872314, max timestamp=1680720872334Num uniques 1112; 2023-04-05 18:54:32,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=11992, suffix heap size=143616,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=262 2023-04-05 18:54:32,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1680720872314, max timestamp=1680720872334Num uniques 1009; 2023-04-05 18:54:32,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46207, new segment data size=11992, suffix heap size=132904,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=262 2023-04-05 18:54:32,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1680720872314, max timestamp=1680720872334Num uniques 1009; 2023-04-05 18:54:32,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46207, new segment data size=11992, suffix heap size=132904,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=262 2023-04-05 18:54:32,350 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/7f808a1aa5dc4672abaed432be8a8c7f 2023-04-05 18:54:32,399 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/090ff72803fc47daad8eb0835c69f9d1 2023-04-05 18:54:32,407 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/7a5b684a6e984128a05b1207f4e30872 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/7a5b684a6e984128a05b1207f4e30872 2023-04-05 18:54:32,413 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/7a5b684a6e984128a05b1207f4e30872, entries=50, sequenceid=436, filesize=11.7 K 2023-04-05 18:54:32,414 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/7f808a1aa5dc4672abaed432be8a8c7f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7f808a1aa5dc4672abaed432be8a8c7f 2023-04-05 18:54:32,427 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7f808a1aa5dc4672abaed432be8a8c7f, entries=50, sequenceid=436, filesize=7.1 K 2023-04-05 18:54:32,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/090ff72803fc47daad8eb0835c69f9d1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/090ff72803fc47daad8eb0835c69f9d1 2023-04-05 18:54:32,433 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/090ff72803fc47daad8eb0835c69f9d1, entries=50, sequenceid=436, filesize=7.1 K 2023-04-05 18:54:32,434 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=147.60 KB/151140 for 5f2134cd982ecee57a32108ad09a562d in 1382ms, sequenceid=436, compaction requested=true 2023-04-05 18:54:32,434 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:32,434 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:32,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:32,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,436 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:32,436 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:32,438 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1680720872381, max timestamp=1680720872436Num uniques 1121; 2023-04-05 18:54:32,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=10116, suffix heap size=144552,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=221 2023-04-05 18:54:32,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1680720872381, max timestamp=1680720872436Num uniques 1121; 2023-04-05 18:54:32,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1680720872381, max timestamp=1680720872436Num uniques 1121; 2023-04-05 18:54:32,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=10116, suffix heap size=144552,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=221 2023-04-05 18:54:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,440 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=33.54 KB heapSize=80.34 KB 2023-04-05 18:54:32,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:32,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=10116, suffix heap size=144552,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=221 2023-04-05 18:54:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:32,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:32,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:32,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:32,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:32,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:32,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:32,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:32,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fbbda38[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:32,442 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:32,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,448 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:32,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:32,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:32,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:32,450 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:32,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:32,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:32,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:32,453 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:32,453 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:32,454 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:32,454 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:32,454 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:32,456 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,457 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,457 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:32,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,458 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:32,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:32,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:32,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:32,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:32,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:32,462 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:32,462 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:32,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:32,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:32,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:32,478 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405a404c93722ac4078bb343034803e305a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a404c93722ac4078bb343034803e305a 2023-04-05 18:54:32,485 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=766, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/70f482f9b8c041c68f3ac717c2d33c4b 2023-04-05 18:54:32,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720872448, max timestamp=1680720872506Num uniques 859; 2023-04-05 18:54:32,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:32,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720872448, max timestamp=1680720872506Num uniques 859; 2023-04-05 18:54:32,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720872448, max timestamp=1680720872506Num uniques 859; 2023-04-05 18:54:32,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:32,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:32,509 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/0625293ef0f24307813874c7ce9fe49a 2023-04-05 18:54:32,562 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:32,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:32,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:32,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:32,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:32,565 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:32,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:32,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:32,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:32,566 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,566 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,568 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:32,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:32,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:32,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:32,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:32,571 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:32,571 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:32,571 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,571 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:32,571 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:32,571 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,571 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:32,571 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:32,571 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,571 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:32,571 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:32,573 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,574 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,574 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,574 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720872551, max timestamp=1680720872573Num uniques 1068; 2023-04-05 18:54:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-05 18:54:32,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:32,576 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:32,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb580cd[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:32,576 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720872551, max timestamp=1680720872573Num uniques 1068; 2023-04-05 18:54:32,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-05 18:54:32,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720872551, max timestamp=1680720872573Num uniques 1068; 2023-04-05 18:54:32,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-05 18:54:32,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:32,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:32,577 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,577 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:32,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:32,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:32,579 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:32,581 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:32,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:32,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:32,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:32,582 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:32,583 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,584 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,584 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:32,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:32,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:32,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:32,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:32,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:32,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:32,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:32,589 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:32,589 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:32,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:32,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720872613, max timestamp=1680720872635Num uniques 1077; 2023-04-05 18:54:32,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720872613, max timestamp=1680720872635Num uniques 1077; 2023-04-05 18:54:32,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-05 18:54:32,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-05 18:54:32,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680720872613, max timestamp=1680720872635Num uniques 1077; 2023-04-05 18:54:32,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-05 18:54:32,689 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:32,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:32,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:32,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:32,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:32,692 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:32,692 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:32,692 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:32,692 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:32,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680720872664, max timestamp=1680720872692Num uniques 1086; 2023-04-05 18:54:32,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-05 18:54:32,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680720872664, max timestamp=1680720872692Num uniques 1086; 2023-04-05 18:54:32,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680720872664, max timestamp=1680720872692Num uniques 1086; 2023-04-05 18:54:32,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-05 18:54:32,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-05 18:54:32,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:32,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:32,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:32,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:32,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:32,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:32,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:32,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:32,698 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,698 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:32,698 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:32,698 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:32,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59782098[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:32,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:32,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:32,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:32,706 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:32,709 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:32,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:32,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:32,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:32,709 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:32,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,711 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:32,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,713 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,716 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:32,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:32,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:32,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:32,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:32,716 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:32,716 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:32,716 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:32,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:32,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680720872751, max timestamp=1680720872763Num uniques 1045; 2023-04-05 18:54:32,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680720872751, max timestamp=1680720872763Num uniques 1045; 2023-04-05 18:54:32,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 18:54:32,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 18:54:32,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680720872751, max timestamp=1680720872763Num uniques 1045; 2023-04-05 18:54:32,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 18:54:32,816 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:32,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:32,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:32,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:32,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:32,819 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:32,819 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:32,819 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:32,820 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:32,820 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,820 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,822 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:32,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:32,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:32,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:32,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:32,824 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:32,824 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:32,824 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:32,824 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:32,824 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:32,824 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,824 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:32,824 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:32,824 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720872803, max timestamp=1680720872824Num uniques 1054; 2023-04-05 18:54:32,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720872803, max timestamp=1680720872824Num uniques 1054; 2023-04-05 18:54:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 18:54:32,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720872803, max timestamp=1680720872824Num uniques 1054; 2023-04-05 18:54:32,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 18:54:32,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 18:54:32,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:32,827 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:32,828 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,829 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,829 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:32,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:32,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43e41450[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:32,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,831 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:32,831 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,831 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:32,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:32,838 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:32,838 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:32,838 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:32,838 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:32,838 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:32,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:32,840 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,842 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:32,844 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:32,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:32,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:32,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:32,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:32,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:32,845 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:32,845 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:32,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:32,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680720872853, max timestamp=1680720872889Num uniques 1013; 2023-04-05 18:54:32,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-05 18:54:32,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680720872853, max timestamp=1680720872889Num uniques 1013; 2023-04-05 18:54:32,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-05 18:54:32,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680720872853, max timestamp=1680720872889Num uniques 1013; 2023-04-05 18:54:32,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-05 18:54:32,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/31c8f33d558f45faa4a5a831b45627c7 2023-04-05 18:54:32,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/70f482f9b8c041c68f3ac717c2d33c4b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/70f482f9b8c041c68f3ac717c2d33c4b 2023-04-05 18:54:32,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/70f482f9b8c041c68f3ac717c2d33c4b, entries=250, sequenceid=766, filesize=39.2 K 2023-04-05 18:54:32,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/0625293ef0f24307813874c7ce9fe49a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0625293ef0f24307813874c7ce9fe49a 2023-04-05 18:54:32,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0625293ef0f24307813874c7ce9fe49a, entries=150, sequenceid=766, filesize=11.9 K 2023-04-05 18:54:32,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/31c8f33d558f45faa4a5a831b45627c7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/31c8f33d558f45faa4a5a831b45627c7 2023-04-05 18:54:32,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/31c8f33d558f45faa4a5a831b45627c7, entries=150, sequenceid=766, filesize=11.9 K 2023-04-05 18:54:32,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.30 KB/82224, currentSize=134.18 KB/137400 for 5f2134cd982ecee57a32108ad09a562d in 500ms, sequenceid=766, compaction requested=true 2023-04-05 18:54:32,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:32,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:32,940 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:54:32,940 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:54:32,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:32,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:32,941 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130804 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:54:32,941 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/A is initiating minor compaction (all files) 2023-04-05 18:54:32,941 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56624 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:54:32,942 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/A in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,942 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/B is initiating minor compaction (all files) 2023-04-05 18:54:32,942 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/B in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,942 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/3610432751204ceda09e0b6865b4da65, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6eac8ad2269d4ed7b774681b667fc092, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/361d5f3d486f425790629897044fc287, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/7a5b684a6e984128a05b1207f4e30872, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/70f482f9b8c041c68f3ac717c2d33c4b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=127.7 K 2023-04-05 18:54:32,942 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ea4efeebaa684ff7bba00ff637389e63, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1fa5a804ed9c44e193d8683e35b7bd73, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a89d095256ca4aca90f23b7c6d742fce, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7f808a1aa5dc4672abaed432be8a8c7f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0625293ef0f24307813874c7ce9fe49a] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=55.3 K 2023-04-05 18:54:32,942 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 3610432751204ceda09e0b6865b4da65, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680720868119 2023-04-05 18:54:32,942 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ea4efeebaa684ff7bba00ff637389e63, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680720868119 2023-04-05 18:54:32,943 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6eac8ad2269d4ed7b774681b667fc092, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680720868255 2023-04-05 18:54:32,943 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1fa5a804ed9c44e193d8683e35b7bd73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680720868255 2023-04-05 18:54:32,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:32,943 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 361d5f3d486f425790629897044fc287, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680720868889 2023-04-05 18:54:32,943 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a89d095256ca4aca90f23b7c6d742fce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680720868889 2023-04-05 18:54:32,943 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=140.89 KB heapSize=399.84 KB 2023-04-05 18:54:32,943 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:32,943 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7a5b684a6e984128a05b1207f4e30872, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680720871050 2023-04-05 18:54:32,943 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 7f808a1aa5dc4672abaed432be8a8c7f, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680720871050 2023-04-05 18:54:32,943 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:32,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:32,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:32,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:32,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:32,944 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 70f482f9b8c041c68f3ac717c2d33c4b, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680720872381 2023-04-05 18:54:32,944 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0625293ef0f24307813874c7ce9fe49a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680720872381 2023-04-05 18:54:32,945 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:32,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:32,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:32,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:32,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:32,949 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:32,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:32,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:32,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:32,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:32,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:32,956 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:32,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:32,958 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:32,958 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:32,958 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:32,960 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,960 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:32,960 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:32,960 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:32,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 767 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720932965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,970 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#B#compaction#181 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:32,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:32,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:32,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,973 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,974 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,974 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226f6c0e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:32,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:32,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,976 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#A#compaction#180 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:32,976 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:32,977 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,977 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:32,978 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:32,978 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:32,978 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,978 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:32,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:32,981 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:32,981 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:32,981 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:32,981 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:32,981 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:32,983 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,983 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,983 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:32,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:32,983 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,985 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:32,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:32,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:32,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:32,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:32,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:32,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:32,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:32,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:32,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:32,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:32,990 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:32,990 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:32,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:32,990 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/a69fec0382284e9aa8b2a1ae048b85de as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a69fec0382284e9aa8b2a1ae048b85de 2023-04-05 18:54:32,995 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/B of 5f2134cd982ecee57a32108ad09a562d into a69fec0382284e9aa8b2a1ae048b85de(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:32,996 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:32,996 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/B, priority=11, startTime=1680720872940; duration=0sec 2023-04-05 18:54:32,996 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:32,996 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:54:32,997 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56624 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:54:32,997 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/C is initiating minor compaction (all files) 2023-04-05 18:54:32,998 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/C in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:32,998 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/179c1350edfa49fa9b2e27ec1033f2cf, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/74ddfc16796842d5bac1fe1dcae1927c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/cc714a157b6d404fb99669e27c04d937, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/090ff72803fc47daad8eb0835c69f9d1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/31c8f33d558f45faa4a5a831b45627c7] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=55.3 K 2023-04-05 18:54:32,998 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 179c1350edfa49fa9b2e27ec1033f2cf, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680720868119 2023-04-05 18:54:32,998 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 74ddfc16796842d5bac1fe1dcae1927c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680720868255 2023-04-05 18:54:33,000 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting cc714a157b6d404fb99669e27c04d937, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680720868889 2023-04-05 18:54:33,000 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 090ff72803fc47daad8eb0835c69f9d1, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680720871050 2023-04-05 18:54:33,001 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 31c8f33d558f45faa4a5a831b45627c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680720872381 2023-04-05 18:54:33,005 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/b036c936ae66495798cc9bf36211e9d7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/b036c936ae66495798cc9bf36211e9d7 2023-04-05 18:54:33,011 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/A of 5f2134cd982ecee57a32108ad09a562d into b036c936ae66495798cc9bf36211e9d7(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:33,012 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:33,012 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/A, priority=11, startTime=1680720872940; duration=0sec 2023-04-05 18:54:33,012 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:33,014 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#C#compaction#182 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:33,031 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/e0556abbfae84b7896b4e126fa69b98d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e0556abbfae84b7896b4e126fa69b98d 2023-04-05 18:54:33,036 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/C of 5f2134cd982ecee57a32108ad09a562d into e0556abbfae84b7896b4e126fa69b98d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:33,037 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:33,037 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/C, priority=11, startTime=1680720872940; duration=0sec 2023-04-05 18:54:33,037 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:33,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:33,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 769 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720933068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,090 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:33,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:33,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:33,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:33,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:33,093 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:33,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:33,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:33,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,096 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,096 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:33,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:33,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:33,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:33,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:33,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:33,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:33,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@324c2366[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:33,103 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:33,103 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,104 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:33,104 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,104 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:33,107 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:33,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:33,110 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:33,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:33,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:33,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:33,110 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:33,111 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,112 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,113 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,116 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,116 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,116 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:33,116 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,116 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:33,116 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,116 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,116 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,116 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,116 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:33,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:33,116 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:33,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,216 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:33,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:33,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:33,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:33,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:33,220 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:33,220 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,220 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:33,220 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:33,220 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,220 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,222 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:33,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:33,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:33,223 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:33,223 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,223 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:33,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:33,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:33,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:33,227 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:33,229 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:33,229 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a28b957[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:33,230 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:33,231 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,231 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:33,232 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,232 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,232 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,232 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:33,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:33,234 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:33,234 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:33,234 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:33,235 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:33,235 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:33,236 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,236 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,236 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,236 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,238 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:33,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:33,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:33,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,243 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,244 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:33,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,244 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:33,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:33,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 771 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720933271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,344 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:33,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:33,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:33,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:33,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:33,348 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:33,348 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,348 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:33,348 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:33,348 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,348 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,349 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,350 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,350 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,350 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:33,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:33,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:33,351 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,351 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:33,351 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:33,351 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:33,353 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,353 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@490d6ed6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:33,355 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:33,355 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,356 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,356 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:33,362 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:33,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:33,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:33,364 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:33,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:33,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:33,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:33,364 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:33,366 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,366 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,366 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,370 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304058699a40766204773b6f2b4fa796ec832 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058699a40766204773b6f2b4fa796ec832 2023-04-05 18:54:33,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,374 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:33,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:33,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:33,375 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:33,375 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:33,376 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=906, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/551c2636b6b84e808f2a0c9aa8872756 2023-04-05 18:54:33,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=906 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/357308a9c4c747eeae0331edc28c9006 2023-04-05 18:54:33,475 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:33,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:33,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:33,478 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:33,478 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,478 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:33,478 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:33,479 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,479 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,480 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,480 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:33,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:33,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:33,482 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,482 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:33,482 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:33,482 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:33,485 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:33,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10db74e5[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:33,486 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:33,487 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,488 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:33,488 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,488 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:33,489 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:33,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:33,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:33,491 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:33,491 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:33,492 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:33,492 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:33,492 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:33,493 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,493 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,493 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,494 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,495 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,498 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,498 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,498 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:33,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:33,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,498 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,498 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,498 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,499 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,499 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,499 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,499 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,499 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:33,500 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:33,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:33,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 773 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720933575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,600 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:33,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:33,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:33,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:33,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:33,603 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:33,603 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,603 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:33,603 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:33,603 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,603 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,604 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,605 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,605 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:33,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:33,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:33,607 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,608 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:33,608 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:33,608 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:33,610 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,610 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,610 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,610 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43df228b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:33,612 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:33,612 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,613 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:33,614 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:33,614 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:33,614 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,614 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:33,616 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,616 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,616 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,616 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:33,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:33,619 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:33,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:33,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:33,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:33,619 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:33,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,623 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,631 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,631 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:33,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:33,631 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,631 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:33,631 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:33,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,731 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:33,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:33,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:33,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:33,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:33,734 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:33,735 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,735 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:33,735 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:33,735 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,735 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,737 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:33,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:33,739 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:33,739 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,739 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:33,739 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:33,739 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:33,739 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,739 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:33,739 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:33,739 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,739 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:33,739 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:33,739 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,739 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:33,739 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:33,741 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c52d9a6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:33,742 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:33,742 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,744 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,744 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:33,744 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:33,744 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:33,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:33,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,745 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:33,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:33,748 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:33,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:33,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:33,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:33,748 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:33,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:33,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:33,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,751 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:33,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,753 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:33,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:33,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:33,755 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:33,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:33,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:33,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:33,755 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:33,756 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:33,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=906 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/c662433630bd4d7bb8c113c00ac447fe 2023-04-05 18:54:33,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/551c2636b6b84e808f2a0c9aa8872756 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/551c2636b6b84e808f2a0c9aa8872756 2023-04-05 18:54:33,809 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/551c2636b6b84e808f2a0c9aa8872756, entries=200, sequenceid=906, filesize=32.3 K 2023-04-05 18:54:33,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/357308a9c4c747eeae0331edc28c9006 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/357308a9c4c747eeae0331edc28c9006 2023-04-05 18:54:33,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/357308a9c4c747eeae0331edc28c9006, entries=150, sequenceid=906, filesize=11.9 K 2023-04-05 18:54:33,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/c662433630bd4d7bb8c113c00ac447fe as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/c662433630bd4d7bb8c113c00ac447fe 2023-04-05 18:54:33,820 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/c662433630bd4d7bb8c113c00ac447fe, entries=150, sequenceid=906, filesize=11.9 K 2023-04-05 18:54:33,821 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.80 KB/409392, currentSize=40.25 KB/41220 for 5f2134cd982ecee57a32108ad09a562d in 878ms, sequenceid=906, compaction requested=false 2023-04-05 18:54:33,821 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:33,856 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:33,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:33,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:33,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:33,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:33,859 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:33,859 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,859 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:33,859 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:33,860 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,860 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,861 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:33,861 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:33,861 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,862 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:33,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:33,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:33,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:33,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:33,862 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:33,862 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:33,862 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:33,863 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:33,863 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:33,863 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:33,863 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. started... 2023-04-05 18:54:33,863 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 18:54:33,863 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:33,863 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:33,863 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:33,863 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:33,863 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:33,863 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:34,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:34,077 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. as already flushing 2023-04-05 18:54:34,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720874105, max timestamp=1680720874116Num uniques 859; 2023-04-05 18:54:34,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720874105, max timestamp=1680720874116Num uniques 859; 2023-04-05 18:54:34,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:34,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:34,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720874105, max timestamp=1680720874116Num uniques 859; 2023-04-05 18:54:34,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:34,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:34,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 809 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720934177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:34,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:34,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 811 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720934280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:34,284 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405b67b0ec4540f4e9fbaa606481e403245 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b67b0ec4540f4e9fbaa606481e403245 2023-04-05 18:54:34,372 DEBUG [Thread-894] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a7b66a3 to 127.0.0.1:49755 2023-04-05 18:54:34,372 DEBUG [Thread-894] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:34,372 DEBUG [Thread-892] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:49755 2023-04-05 18:54:34,373 DEBUG [Thread-892] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:34,374 DEBUG [Thread-890] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:49755 2023-04-05 18:54:34,374 DEBUG [Thread-890] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:34,376 DEBUG [Thread-896] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42d325f4 to 127.0.0.1:49755 2023-04-05 18:54:34,376 DEBUG [Thread-896] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:34,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:34,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 813 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720934483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:34,691 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=918, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/72c881e29dfd43b3b35f18c479cd6a6d 2023-04-05 18:54:34,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:34,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 815 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720934784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,101 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/e2b6e6b8bd154aff9726a87b586b6151 2023-04-05 18:54:35,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:35,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 817 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55836 deadline: 1680720935287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,511 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/21bb0e4c030146c6a034087c320cf85b 2023-04-05 18:54:35,515 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/72c881e29dfd43b3b35f18c479cd6a6d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/72c881e29dfd43b3b35f18c479cd6a6d 2023-04-05 18:54:35,519 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/72c881e29dfd43b3b35f18c479cd6a6d, entries=150, sequenceid=918, filesize=25.4 K 2023-04-05 18:54:35,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/e2b6e6b8bd154aff9726a87b586b6151 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e2b6e6b8bd154aff9726a87b586b6151 2023-04-05 18:54:35,523 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e2b6e6b8bd154aff9726a87b586b6151, entries=150, sequenceid=918, filesize=11.9 K 2023-04-05 18:54:35,524 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/21bb0e4c030146c6a034087c320cf85b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/21bb0e4c030146c6a034087c320cf85b 2023-04-05 18:54:35,527 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/21bb0e4c030146c6a034087c320cf85b, entries=150, sequenceid=918, filesize=11.9 K 2023-04-05 18:54:35,528 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=138.65 KB/141980 for 5f2134cd982ecee57a32108ad09a562d in 1665ms, sequenceid=918, compaction requested=true 2023-04-05 18:54:35,528 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:35,528 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:35,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:35,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:35,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:35,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:35,529 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:35,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:35,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e0b6924[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:35,531 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:35,531 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:35,532 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:35,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:35,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:35,532 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,532 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:35,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:35,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:35,533 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:35,533 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,533 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:35,533 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:35,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:35,535 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:35,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:35,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:35,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:35,535 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:35,540 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:35,540 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:35,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:35,540 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:35,540 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:35,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:35,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:35,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:35,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:35,543 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:35,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:35,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:35,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:35,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:35,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:35,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:35,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:35,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:35,543 DEBUG [Thread-889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:35,543 DEBUG [Thread-889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:35,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:35,643 DEBUG [Thread-889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:35,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:36,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:36,297 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=145.36 KB heapSize=415.77 KB 2023-04-05 18:54:36,297 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:36,297 DEBUG [Thread-881] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:49755 2023-04-05 18:54:36,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:36,298 DEBUG [Thread-881] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:36,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:36,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:36,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:36,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:36,710 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405e6c9ba0e8f0a4f0cbcd9ad8b4b343e61 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e6c9ba0e8f0a4f0cbcd9ad8b4b343e61 2023-04-05 18:54:36,736 DEBUG [Thread-883] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:49755 2023-04-05 18:54:36,737 DEBUG [Thread-883] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:36,758 DEBUG [Thread-885] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:49755 2023-04-05 18:54:36,758 DEBUG [Thread-885] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:37,115 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=956, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/9cbaaa72b1a94fd4a4a5d761427b2bd7 2023-04-05 18:54:37,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/0be0c12e53fd481c84b9c38f0201dc3c 2023-04-05 18:54:37,863 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-05 18:54:37,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/233c89b5eb794b21bd52c2ec25243832 2023-04-05 18:54:37,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/9cbaaa72b1a94fd4a4a5d761427b2bd7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/9cbaaa72b1a94fd4a4a5d761427b2bd7 2023-04-05 18:54:37,944 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/9cbaaa72b1a94fd4a4a5d761427b2bd7, entries=150, sequenceid=956, filesize=25.4 K 2023-04-05 18:54:37,945 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/0be0c12e53fd481c84b9c38f0201dc3c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0be0c12e53fd481c84b9c38f0201dc3c 2023-04-05 18:54:37,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0be0c12e53fd481c84b9c38f0201dc3c, entries=150, sequenceid=956, filesize=11.9 K 2023-04-05 18:54:37,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/233c89b5eb794b21bd52c2ec25243832 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/233c89b5eb794b21bd52c2ec25243832 2023-04-05 18:54:37,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/233c89b5eb794b21bd52c2ec25243832, entries=150, sequenceid=956, filesize=11.9 K 2023-04-05 18:54:37,953 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~145.36 KB/148850, heapSize ~415.72 KB/425696, currentSize=13.42 KB/13740 for 5f2134cd982ecee57a32108ad09a562d in 1656ms, sequenceid=956, compaction requested=true 2023-04-05 18:54:37,953 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:37,953 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:37,953 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:37,953 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:37,953 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:54:37,953 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:54:37,955 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:54:37,955 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/B is initiating minor compaction (all files) 2023-04-05 18:54:37,955 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111974 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:54:37,955 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/B in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:37,955 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/A is initiating minor compaction (all files) 2023-04-05 18:54:37,955 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a69fec0382284e9aa8b2a1ae048b85de, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/357308a9c4c747eeae0331edc28c9006, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e2b6e6b8bd154aff9726a87b586b6151, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0be0c12e53fd481c84b9c38f0201dc3c] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=48.3 K 2023-04-05 18:54:37,955 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/A in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:37,955 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/b036c936ae66495798cc9bf36211e9d7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/551c2636b6b84e808f2a0c9aa8872756, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/72c881e29dfd43b3b35f18c479cd6a6d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/9cbaaa72b1a94fd4a4a5d761427b2bd7] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=109.3 K 2023-04-05 18:54:37,955 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a69fec0382284e9aa8b2a1ae048b85de, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680720872381 2023-04-05 18:54:37,955 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting b036c936ae66495798cc9bf36211e9d7, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680720872381 2023-04-05 18:54:37,956 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 357308a9c4c747eeae0331edc28c9006, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=906, earliestPutTs=1680720872932 2023-04-05 18:54:37,956 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 551c2636b6b84e808f2a0c9aa8872756, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=906, earliestPutTs=1680720872930 2023-04-05 18:54:37,956 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e2b6e6b8bd154aff9726a87b586b6151, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=918, earliestPutTs=1680720872955 2023-04-05 18:54:37,956 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 72c881e29dfd43b3b35f18c479cd6a6d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=918, earliestPutTs=1680720872955 2023-04-05 18:54:37,956 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0be0c12e53fd481c84b9c38f0201dc3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1680720874167 2023-04-05 18:54:37,956 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 9cbaaa72b1a94fd4a4a5d761427b2bd7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1680720874167 2023-04-05 18:54:37,972 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#B#compaction#192 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:37,975 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#A#compaction#191 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:38,382 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/476f47e02e4a4e93a8048560fe3337a4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/476f47e02e4a4e93a8048560fe3337a4 2023-04-05 18:54:38,387 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/B of 5f2134cd982ecee57a32108ad09a562d into 476f47e02e4a4e93a8048560fe3337a4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:38,387 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:38,387 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/B, priority=12, startTime=1680720877953; duration=0sec 2023-04-05 18:54:38,387 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:38,387 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:54:38,388 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:54:38,388 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 5f2134cd982ecee57a32108ad09a562d/C is initiating minor compaction (all files) 2023-04-05 18:54:38,388 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5f2134cd982ecee57a32108ad09a562d/C in TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:38,388 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e0556abbfae84b7896b4e126fa69b98d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/c662433630bd4d7bb8c113c00ac447fe, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/21bb0e4c030146c6a034087c320cf85b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/233c89b5eb794b21bd52c2ec25243832] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp, totalSize=48.3 K 2023-04-05 18:54:38,389 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e0556abbfae84b7896b4e126fa69b98d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680720872381 2023-04-05 18:54:38,389 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting c662433630bd4d7bb8c113c00ac447fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=906, earliestPutTs=1680720872932 2023-04-05 18:54:38,389 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 21bb0e4c030146c6a034087c320cf85b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=918, earliestPutTs=1680720872955 2023-04-05 18:54:38,389 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 233c89b5eb794b21bd52c2ec25243832, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1680720874167 2023-04-05 18:54:38,396 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 5f2134cd982ecee57a32108ad09a562d#C#compaction#193 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:38,805 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/c205947970b64b9b98630df0ae581fcc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/c205947970b64b9b98630df0ae581fcc 2023-04-05 18:54:38,809 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/C of 5f2134cd982ecee57a32108ad09a562d into c205947970b64b9b98630df0ae581fcc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:38,809 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:38,809 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/C, priority=12, startTime=1680720877953; duration=0sec 2023-04-05 18:54:38,809 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:39,195 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/154f14641228476ab331842f35d3b44c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/154f14641228476ab331842f35d3b44c 2023-04-05 18:54:39,199 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5f2134cd982ecee57a32108ad09a562d/A of 5f2134cd982ecee57a32108ad09a562d into 154f14641228476ab331842f35d3b44c(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 18:54:39,199 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:39,199 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d., storeName=5f2134cd982ecee57a32108ad09a562d/A, priority=12, startTime=1680720877953; duration=1sec 2023-04-05 18:54:39,199 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:39,820 DEBUG [Thread-879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55cbb115 to 127.0.0.1:49755 2023-04-05 18:54:39,820 DEBUG [Thread-879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:39,880 DEBUG [Thread-887] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:49755 2023-04-05 18:54:39,880 DEBUG [Thread-887] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 716 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 34 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 5123 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 4599 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 2051 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 6151 rows 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 2053 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 6158 rows 2023-04-05 18:54:39,880 INFO [Listener at localhost/43705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 18:54:39,881 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x324dc814 to 127.0.0.1:49755 2023-04-05 18:54:39,881 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:54:39,882 INFO [Listener at localhost/43705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 18:54:39,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 18:54:39,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:39,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 18:54:39,908 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720879908"}]},"ts":"1680720879908"} 2023-04-05 18:54:39,909 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 18:54:39,912 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 18:54:39,913 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, UNASSIGN}] 2023-04-05 18:54:39,914 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, UNASSIGN 2023-04-05 18:54:39,916 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=5f2134cd982ecee57a32108ad09a562d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:39,916 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720879916"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720879916"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720879916"}]},"ts":"1680720879916"} 2023-04-05 18:54:39,917 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:54:40,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 18:54:40,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5f2134cd982ecee57a32108ad09a562d, disabling compactions & flushes 2023-04-05 18:54:40,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. after waiting 0 ms 2023-04-05 18:54:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:40,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5f2134cd982ecee57a32108ad09a562d 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-05 18:54:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=A 2023-04-05 18:54:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=B 2023-04-05 18:54:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5f2134cd982ecee57a32108ad09a562d, store=C 2023-04-05 18:54:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:40,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 18:54:40,480 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304055fad02c706c340248dacb7b772a79f82 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055fad02c706c340248dacb7b772a79f82 2023-04-05 18:54:40,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 18:54:40,886 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=966, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/7b271b6260754a0887f993da12ae1334 2023-04-05 18:54:41,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 18:54:41,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=966 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/59ad9002ee5a4d84bffe1bc6fd12c826 2023-04-05 18:54:41,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=966 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/5d83ad02510b496a8327a52f30c21138 2023-04-05 18:54:41,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/A/7b271b6260754a0887f993da12ae1334 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/7b271b6260754a0887f993da12ae1334 2023-04-05 18:54:41,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/7b271b6260754a0887f993da12ae1334, entries=100, sequenceid=966, filesize=18.6 K 2023-04-05 18:54:41,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/B/59ad9002ee5a4d84bffe1bc6fd12c826 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/59ad9002ee5a4d84bffe1bc6fd12c826 2023-04-05 18:54:41,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/59ad9002ee5a4d84bffe1bc6fd12c826, entries=100, sequenceid=966, filesize=9.5 K 2023-04-05 18:54:41,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/.tmp/C/5d83ad02510b496a8327a52f30c21138 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/5d83ad02510b496a8327a52f30c21138 2023-04-05 18:54:41,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/5d83ad02510b496a8327a52f30c21138, entries=100, sequenceid=966, filesize=9.5 K 2023-04-05 18:54:41,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 5f2134cd982ecee57a32108ad09a562d in 1652ms, sequenceid=966, compaction requested=false 2023-04-05 18:54:41,742 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/381d324829664cb59286483366ad4b5f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/abb3727124de48e983a1275a69895be6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/d9dfecf59dec45448dbadf026bbd339f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a85fca9fea22493a8c97df850c9cd3c3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6f95db6cc0f745c699b4dfed2c7e0b9a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/41f7d25ca9654c8e875f13e1afb87311, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/c1360495e21d450094dd52f9304f2a0d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5ebcfbb56cba4d60a7e42f327f6aafc7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/fd79a13abcce4fef97405bfe9406fd02, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/e5a944ac0fcd4bab868c4ee0c26e53fb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5604ba7064494a83a4685f3a766a1a58, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/4b3d4f0446d34b51b4bce91a1b283322, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a0828ca531274532bf399055ee4902e1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/21c3002db287487aab1c8a38a2d09e7e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a72b92a6528a43b5986dd7cc5a2443e7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a2310e3324c04f188c7fe2efd7764258, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/1ceb03ee2a0340c89609c595868f3343, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/3610432751204ceda09e0b6865b4da65, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6eac8ad2269d4ed7b774681b667fc092, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/361d5f3d486f425790629897044fc287, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/7a5b684a6e984128a05b1207f4e30872, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/70f482f9b8c041c68f3ac717c2d33c4b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/b036c936ae66495798cc9bf36211e9d7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/551c2636b6b84e808f2a0c9aa8872756, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/72c881e29dfd43b3b35f18c479cd6a6d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/9cbaaa72b1a94fd4a4a5d761427b2bd7] to archive 2023-04-05 18:54:41,743 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:54:41,745 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/381d324829664cb59286483366ad4b5f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/381d324829664cb59286483366ad4b5f 2023-04-05 18:54:41,746 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/abb3727124de48e983a1275a69895be6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/abb3727124de48e983a1275a69895be6 2023-04-05 18:54:41,747 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/d9dfecf59dec45448dbadf026bbd339f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/d9dfecf59dec45448dbadf026bbd339f 2023-04-05 18:54:41,748 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a85fca9fea22493a8c97df850c9cd3c3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a85fca9fea22493a8c97df850c9cd3c3 2023-04-05 18:54:41,749 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6f95db6cc0f745c699b4dfed2c7e0b9a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6f95db6cc0f745c699b4dfed2c7e0b9a 2023-04-05 18:54:41,750 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/41f7d25ca9654c8e875f13e1afb87311 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/41f7d25ca9654c8e875f13e1afb87311 2023-04-05 18:54:41,751 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/c1360495e21d450094dd52f9304f2a0d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/c1360495e21d450094dd52f9304f2a0d 2023-04-05 18:54:41,752 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5ebcfbb56cba4d60a7e42f327f6aafc7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5ebcfbb56cba4d60a7e42f327f6aafc7 2023-04-05 18:54:41,753 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/fd79a13abcce4fef97405bfe9406fd02 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/fd79a13abcce4fef97405bfe9406fd02 2023-04-05 18:54:41,754 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/e5a944ac0fcd4bab868c4ee0c26e53fb to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/e5a944ac0fcd4bab868c4ee0c26e53fb 2023-04-05 18:54:41,755 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5604ba7064494a83a4685f3a766a1a58 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/5604ba7064494a83a4685f3a766a1a58 2023-04-05 18:54:41,756 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/4b3d4f0446d34b51b4bce91a1b283322 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/4b3d4f0446d34b51b4bce91a1b283322 2023-04-05 18:54:41,757 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a0828ca531274532bf399055ee4902e1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a0828ca531274532bf399055ee4902e1 2023-04-05 18:54:41,758 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/21c3002db287487aab1c8a38a2d09e7e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/21c3002db287487aab1c8a38a2d09e7e 2023-04-05 18:54:41,759 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a72b92a6528a43b5986dd7cc5a2443e7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a72b92a6528a43b5986dd7cc5a2443e7 2023-04-05 18:54:41,761 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a2310e3324c04f188c7fe2efd7764258 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/a2310e3324c04f188c7fe2efd7764258 2023-04-05 18:54:41,762 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/1ceb03ee2a0340c89609c595868f3343 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/1ceb03ee2a0340c89609c595868f3343 2023-04-05 18:54:41,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/3610432751204ceda09e0b6865b4da65 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/3610432751204ceda09e0b6865b4da65 2023-04-05 18:54:41,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6eac8ad2269d4ed7b774681b667fc092 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/6eac8ad2269d4ed7b774681b667fc092 2023-04-05 18:54:41,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/361d5f3d486f425790629897044fc287 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/361d5f3d486f425790629897044fc287 2023-04-05 18:54:41,767 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/7a5b684a6e984128a05b1207f4e30872 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/7a5b684a6e984128a05b1207f4e30872 2023-04-05 18:54:41,768 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/70f482f9b8c041c68f3ac717c2d33c4b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/70f482f9b8c041c68f3ac717c2d33c4b 2023-04-05 18:54:41,770 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/b036c936ae66495798cc9bf36211e9d7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/b036c936ae66495798cc9bf36211e9d7 2023-04-05 18:54:41,771 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/551c2636b6b84e808f2a0c9aa8872756 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/551c2636b6b84e808f2a0c9aa8872756 2023-04-05 18:54:41,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/72c881e29dfd43b3b35f18c479cd6a6d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/72c881e29dfd43b3b35f18c479cd6a6d 2023-04-05 18:54:41,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/9cbaaa72b1a94fd4a4a5d761427b2bd7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/9cbaaa72b1a94fd4a4a5d761427b2bd7 2023-04-05 18:54:41,776 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/eab9407073ce49fdb0af5c42b3f69f14, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/94e5db6517354152b1a6bf77ab7bcd37, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/4a68f82768224e099bfd7e23fa80eec6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1b7ead8654e64fb9b1fdd3acf94d0ec8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/9caf8f817694420c923f7ed39ccf5f6f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/3a2a12e828974259b517a3ab1e603a99, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/674bbcdd9b9c4a9486350b962d90870d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ed8ee66c9c3d40019e1a6702052fd553, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/77930071eefe45ae99af88b3a571b37a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/23363e235bcf4fbcaf1fea63d1838ff6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/297022b677ce4bc58adc5f92732c3ba1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e4145cbf638646c88abf4818db25b9e0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/5fc1378f5cee4c05a6126537f63b3e1e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/31cc92b239e848d5bdb1f004b9ef9ed6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a1f76fd11b1f449f95cd278ea0652ef8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/73847a894a3342e781210bdc532b9959, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ea4efeebaa684ff7bba00ff637389e63, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7e7424971e534d249b072fc6a470f839, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1fa5a804ed9c44e193d8683e35b7bd73, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a89d095256ca4aca90f23b7c6d742fce, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7f808a1aa5dc4672abaed432be8a8c7f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a69fec0382284e9aa8b2a1ae048b85de, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0625293ef0f24307813874c7ce9fe49a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/357308a9c4c747eeae0331edc28c9006, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e2b6e6b8bd154aff9726a87b586b6151, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0be0c12e53fd481c84b9c38f0201dc3c] to archive 2023-04-05 18:54:41,777 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:54:41,779 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/eab9407073ce49fdb0af5c42b3f69f14 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/eab9407073ce49fdb0af5c42b3f69f14 2023-04-05 18:54:41,780 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/94e5db6517354152b1a6bf77ab7bcd37 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/94e5db6517354152b1a6bf77ab7bcd37 2023-04-05 18:54:41,782 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/4a68f82768224e099bfd7e23fa80eec6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/4a68f82768224e099bfd7e23fa80eec6 2023-04-05 18:54:41,783 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1b7ead8654e64fb9b1fdd3acf94d0ec8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1b7ead8654e64fb9b1fdd3acf94d0ec8 2023-04-05 18:54:41,784 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/9caf8f817694420c923f7ed39ccf5f6f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/9caf8f817694420c923f7ed39ccf5f6f 2023-04-05 18:54:41,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/3a2a12e828974259b517a3ab1e603a99 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/3a2a12e828974259b517a3ab1e603a99 2023-04-05 18:54:41,786 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/674bbcdd9b9c4a9486350b962d90870d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/674bbcdd9b9c4a9486350b962d90870d 2023-04-05 18:54:41,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ed8ee66c9c3d40019e1a6702052fd553 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ed8ee66c9c3d40019e1a6702052fd553 2023-04-05 18:54:41,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/77930071eefe45ae99af88b3a571b37a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/77930071eefe45ae99af88b3a571b37a 2023-04-05 18:54:41,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/23363e235bcf4fbcaf1fea63d1838ff6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/23363e235bcf4fbcaf1fea63d1838ff6 2023-04-05 18:54:41,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/297022b677ce4bc58adc5f92732c3ba1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/297022b677ce4bc58adc5f92732c3ba1 2023-04-05 18:54:41,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e4145cbf638646c88abf4818db25b9e0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e4145cbf638646c88abf4818db25b9e0 2023-04-05 18:54:41,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/5fc1378f5cee4c05a6126537f63b3e1e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/5fc1378f5cee4c05a6126537f63b3e1e 2023-04-05 18:54:41,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/31cc92b239e848d5bdb1f004b9ef9ed6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/31cc92b239e848d5bdb1f004b9ef9ed6 2023-04-05 18:54:41,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a1f76fd11b1f449f95cd278ea0652ef8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a1f76fd11b1f449f95cd278ea0652ef8 2023-04-05 18:54:41,797 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/73847a894a3342e781210bdc532b9959 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/73847a894a3342e781210bdc532b9959 2023-04-05 18:54:41,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ea4efeebaa684ff7bba00ff637389e63 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/ea4efeebaa684ff7bba00ff637389e63 2023-04-05 18:54:41,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7e7424971e534d249b072fc6a470f839 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7e7424971e534d249b072fc6a470f839 2023-04-05 18:54:41,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1fa5a804ed9c44e193d8683e35b7bd73 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/1fa5a804ed9c44e193d8683e35b7bd73 2023-04-05 18:54:41,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a89d095256ca4aca90f23b7c6d742fce to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a89d095256ca4aca90f23b7c6d742fce 2023-04-05 18:54:41,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7f808a1aa5dc4672abaed432be8a8c7f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/7f808a1aa5dc4672abaed432be8a8c7f 2023-04-05 18:54:41,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a69fec0382284e9aa8b2a1ae048b85de to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/a69fec0382284e9aa8b2a1ae048b85de 2023-04-05 18:54:41,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0625293ef0f24307813874c7ce9fe49a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0625293ef0f24307813874c7ce9fe49a 2023-04-05 18:54:41,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/357308a9c4c747eeae0331edc28c9006 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/357308a9c4c747eeae0331edc28c9006 2023-04-05 18:54:41,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e2b6e6b8bd154aff9726a87b586b6151 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/e2b6e6b8bd154aff9726a87b586b6151 2023-04-05 18:54:41,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0be0c12e53fd481c84b9c38f0201dc3c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/0be0c12e53fd481c84b9c38f0201dc3c 2023-04-05 18:54:41,810 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1a5e850f4fdc441899e318f62389f5ed, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e8dda066b9104e9f90f3ef294d4f41a0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/3c674b3e402f4297b9926c71e34245d5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/bd007bbd0d194acdb8a5a5e24280d0db, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/89dec585cfc84e47b10bc2503871ec2b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/5e41949dcd954c52b636c8d145afc77e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a614455a1b6a4425ae4f0e0d51eaf039, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/dbfef999b29a4bdabff2cef56cd8b94b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a80ac4372444416d817067f6c350d45d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/893e22f04f734714a34af5ca89654a9c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/26d8906213d648ec87c3fd0bcfd5148c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1d1e6040c5f847efb31fc05b977fbd46, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e6cc5486b4ef42bb905a2294fff922c0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/ca1babfdc7b74b488b4597c30b576f57, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/df64313ae19e4b46a626621f93a9e846, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/4ffde19f034e43aca6970b7fe2219b8b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/179c1350edfa49fa9b2e27ec1033f2cf, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/59c058345f794e7e8eda2bbf37751018, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/74ddfc16796842d5bac1fe1dcae1927c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/cc714a157b6d404fb99669e27c04d937, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/090ff72803fc47daad8eb0835c69f9d1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e0556abbfae84b7896b4e126fa69b98d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/31c8f33d558f45faa4a5a831b45627c7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/c662433630bd4d7bb8c113c00ac447fe, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/21bb0e4c030146c6a034087c320cf85b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/233c89b5eb794b21bd52c2ec25243832] to archive 2023-04-05 18:54:41,810 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:54:41,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1a5e850f4fdc441899e318f62389f5ed to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1a5e850f4fdc441899e318f62389f5ed 2023-04-05 18:54:41,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e8dda066b9104e9f90f3ef294d4f41a0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e8dda066b9104e9f90f3ef294d4f41a0 2023-04-05 18:54:41,815 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/3c674b3e402f4297b9926c71e34245d5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/3c674b3e402f4297b9926c71e34245d5 2023-04-05 18:54:41,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/bd007bbd0d194acdb8a5a5e24280d0db to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/bd007bbd0d194acdb8a5a5e24280d0db 2023-04-05 18:54:41,818 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/89dec585cfc84e47b10bc2503871ec2b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/89dec585cfc84e47b10bc2503871ec2b 2023-04-05 18:54:41,819 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/5e41949dcd954c52b636c8d145afc77e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/5e41949dcd954c52b636c8d145afc77e 2023-04-05 18:54:41,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a614455a1b6a4425ae4f0e0d51eaf039 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a614455a1b6a4425ae4f0e0d51eaf039 2023-04-05 18:54:41,822 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/dbfef999b29a4bdabff2cef56cd8b94b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/dbfef999b29a4bdabff2cef56cd8b94b 2023-04-05 18:54:41,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a80ac4372444416d817067f6c350d45d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/a80ac4372444416d817067f6c350d45d 2023-04-05 18:54:41,825 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/893e22f04f734714a34af5ca89654a9c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/893e22f04f734714a34af5ca89654a9c 2023-04-05 18:54:41,826 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/26d8906213d648ec87c3fd0bcfd5148c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/26d8906213d648ec87c3fd0bcfd5148c 2023-04-05 18:54:41,827 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1d1e6040c5f847efb31fc05b977fbd46 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/1d1e6040c5f847efb31fc05b977fbd46 2023-04-05 18:54:41,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e6cc5486b4ef42bb905a2294fff922c0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e6cc5486b4ef42bb905a2294fff922c0 2023-04-05 18:54:41,830 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/ca1babfdc7b74b488b4597c30b576f57 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/ca1babfdc7b74b488b4597c30b576f57 2023-04-05 18:54:41,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/df64313ae19e4b46a626621f93a9e846 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/df64313ae19e4b46a626621f93a9e846 2023-04-05 18:54:41,832 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/4ffde19f034e43aca6970b7fe2219b8b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/4ffde19f034e43aca6970b7fe2219b8b 2023-04-05 18:54:41,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/179c1350edfa49fa9b2e27ec1033f2cf to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/179c1350edfa49fa9b2e27ec1033f2cf 2023-04-05 18:54:41,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/59c058345f794e7e8eda2bbf37751018 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/59c058345f794e7e8eda2bbf37751018 2023-04-05 18:54:41,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/74ddfc16796842d5bac1fe1dcae1927c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/74ddfc16796842d5bac1fe1dcae1927c 2023-04-05 18:54:41,836 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/cc714a157b6d404fb99669e27c04d937 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/cc714a157b6d404fb99669e27c04d937 2023-04-05 18:54:41,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/090ff72803fc47daad8eb0835c69f9d1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/090ff72803fc47daad8eb0835c69f9d1 2023-04-05 18:54:41,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e0556abbfae84b7896b4e126fa69b98d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/e0556abbfae84b7896b4e126fa69b98d 2023-04-05 18:54:41,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/31c8f33d558f45faa4a5a831b45627c7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/31c8f33d558f45faa4a5a831b45627c7 2023-04-05 18:54:41,841 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/c662433630bd4d7bb8c113c00ac447fe to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/c662433630bd4d7bb8c113c00ac447fe 2023-04-05 18:54:41,843 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/21bb0e4c030146c6a034087c320cf85b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/21bb0e4c030146c6a034087c320cf85b 2023-04-05 18:54:41,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/233c89b5eb794b21bd52c2ec25243832 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/233c89b5eb794b21bd52c2ec25243832 2023-04-05 18:54:41,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/recovered.edits/969.seqid, newMaxSeqId=969, maxSeqId=4 2023-04-05 18:54:41,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d. 2023-04-05 18:54:41,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5f2134cd982ecee57a32108ad09a562d: 2023-04-05 18:54:41,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:41,853 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=5f2134cd982ecee57a32108ad09a562d, regionState=CLOSED 2023-04-05 18:54:41,853 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720881853"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720881853"}]},"ts":"1680720881853"} 2023-04-05 18:54:41,855 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-05 18:54:41,855 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 5f2134cd982ecee57a32108ad09a562d, server=jenkins-hbase4.apache.org,33489,1680720816595 in 1.9370 sec 2023-04-05 18:54:41,857 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-05 18:54:41,857 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5f2134cd982ecee57a32108ad09a562d, UNASSIGN in 1.9420 sec 2023-04-05 18:54:41,857 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720881857"}]},"ts":"1680720881857"} 2023-04-05 18:54:41,858 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 18:54:41,860 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 18:54:41,861 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9770 sec 2023-04-05 18:54:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 18:54:42,011 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-05 18:54:42,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 18:54:42,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:42,014 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:42,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-05 18:54:42,015 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:42,018 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:42,021 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/recovered.edits] 2023-04-05 18:54:42,024 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/154f14641228476ab331842f35d3b44c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/154f14641228476ab331842f35d3b44c 2023-04-05 18:54:42,025 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/7b271b6260754a0887f993da12ae1334 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/A/7b271b6260754a0887f993da12ae1334 2023-04-05 18:54:42,027 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/476f47e02e4a4e93a8048560fe3337a4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/476f47e02e4a4e93a8048560fe3337a4 2023-04-05 18:54:42,029 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/59ad9002ee5a4d84bffe1bc6fd12c826 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/B/59ad9002ee5a4d84bffe1bc6fd12c826 2023-04-05 18:54:42,031 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/5d83ad02510b496a8327a52f30c21138 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/5d83ad02510b496a8327a52f30c21138 2023-04-05 18:54:42,032 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/c205947970b64b9b98630df0ae581fcc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/C/c205947970b64b9b98630df0ae581fcc 2023-04-05 18:54:42,035 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/recovered.edits/969.seqid to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d/recovered.edits/969.seqid 2023-04-05 18:54:42,036 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/5f2134cd982ecee57a32108ad09a562d 2023-04-05 18:54:42,036 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:54:42,037 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 18:54:42,037 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-05 18:54:42,041 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050400633465c04bb98b2274567880045e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050400633465c04bb98b2274567880045e 2023-04-05 18:54:42,042 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050f11ebcf58a148e396e63391a630b577 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050f11ebcf58a148e396e63391a630b577 2023-04-05 18:54:42,043 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051b0e52125136480eb708f7bfec2006a2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051b0e52125136480eb708f7bfec2006a2 2023-04-05 18:54:42,044 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051fec2d0050944046a0332816d3e440a5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051fec2d0050944046a0332816d3e440a5 2023-04-05 18:54:42,045 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040536cbe510dedf482a920b4fd4165bd360 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040536cbe510dedf482a920b4fd4165bd360 2023-04-05 18:54:42,046 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405395f7e9d68894ce1bc19a2e9780c5847 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405395f7e9d68894ce1bc19a2e9780c5847 2023-04-05 18:54:42,048 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040542bfb2936eae45f982c92e635fa3b0be to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040542bfb2936eae45f982c92e635fa3b0be 2023-04-05 18:54:42,049 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055173f58faa7a49b2b9646cd3a79b79a4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055173f58faa7a49b2b9646cd3a79b79a4 2023-04-05 18:54:42,050 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040552df13d78e8e49efb235181af45a6f61 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040552df13d78e8e49efb235181af45a6f61 2023-04-05 18:54:42,051 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055fad02c706c340248dacb7b772a79f82 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055fad02c706c340248dacb7b772a79f82 2023-04-05 18:54:42,052 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405826e9c778a554e849af4f873405a7b51 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405826e9c778a554e849af4f873405a7b51 2023-04-05 18:54:42,053 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058699a40766204773b6f2b4fa796ec832 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058699a40766204773b6f2b4fa796ec832 2023-04-05 18:54:42,054 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040594dbb50dd02844c1ae7a97798687abd3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040594dbb50dd02844c1ae7a97798687abd3 2023-04-05 18:54:42,055 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a300526bf29549da90f878fdd29c235a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a300526bf29549da90f878fdd29c235a 2023-04-05 18:54:42,056 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a404c93722ac4078bb343034803e305a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a404c93722ac4078bb343034803e305a 2023-04-05 18:54:42,057 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ae0948b4f6b14e82bd72a7fe29d33fd6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ae0948b4f6b14e82bd72a7fe29d33fd6 2023-04-05 18:54:42,058 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b67b0ec4540f4e9fbaa606481e403245 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b67b0ec4540f4e9fbaa606481e403245 2023-04-05 18:54:42,059 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cdd5f81c90294b559b44812ba9d7f844 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cdd5f81c90294b559b44812ba9d7f844 2023-04-05 18:54:42,060 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d9f90a2aec52410c98b13b641b73ec68 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d9f90a2aec52410c98b13b641b73ec68 2023-04-05 18:54:42,061 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405de94920695e14a21b75b72090ef3d42e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405de94920695e14a21b75b72090ef3d42e 2023-04-05 18:54:42,062 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e6c9ba0e8f0a4f0cbcd9ad8b4b343e61 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e6c9ba0e8f0a4f0cbcd9ad8b4b343e61 2023-04-05 18:54:42,064 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ed7475dbdfed4fc28ce55d4bf813d711 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ed7475dbdfed4fc28ce55d4bf813d711 2023-04-05 18:54:42,064 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 18:54:42,066 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:42,068 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 18:54:42,070 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 18:54:42,071 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:42,071 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 18:54:42,071 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680720882071"}]},"ts":"9223372036854775807"} 2023-04-05 18:54:42,073 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 18:54:42,073 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5f2134cd982ecee57a32108ad09a562d, NAME => 'TestAcidGuarantees,,1680720851651.5f2134cd982ecee57a32108ad09a562d.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 18:54:42,073 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 18:54:42,073 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680720882073"}]},"ts":"9223372036854775807"} 2023-04-05 18:54:42,074 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 18:54:42,076 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:42,077 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-04-05 18:54:42,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-05 18:54:42,115 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-05 18:54:42,127 INFO [Listener at localhost/43705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=225 (was 230), OpenFileDescriptor=537 (was 551), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=360 (was 325) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1419 (was 1724) 2023-04-05 18:54:42,138 INFO [Listener at localhost/43705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=225, OpenFileDescriptor=537, MaxFileDescriptor=60000, SystemLoadAverage=360, ProcessCount=169, AvailableMemoryMB=1418 2023-04-05 18:54:42,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 18:54:42,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 18:54:42,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 18:54:42,143 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 18:54:42,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-05 18:54:42,144 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 18:54:42,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 18:54:42,145 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:42,146 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290 empty. 2023-04-05 18:54:42,146 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:42,146 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:54:42,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 18:54:42,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 18:54:42,554 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 18:54:42,555 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c2dced816e56beff0d38901aab0de290, NAME => 'TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp 2023-04-05 18:54:42,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 18:54:42,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:54:42,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c2dced816e56beff0d38901aab0de290, disabling compactions & flushes 2023-04-05 18:54:42,961 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:42,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:42,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. after waiting 0 ms 2023-04-05 18:54:42,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:42,961 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:42,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:42,963 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 18:54:42,964 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720882964"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720882964"}]},"ts":"1680720882964"} 2023-04-05 18:54:42,965 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 18:54:42,966 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 18:54:42,966 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720882966"}]},"ts":"1680720882966"} 2023-04-05 18:54:42,967 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 18:54:42,973 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2dced816e56beff0d38901aab0de290, ASSIGN}] 2023-04-05 18:54:42,974 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2dced816e56beff0d38901aab0de290, ASSIGN 2023-04-05 18:54:42,975 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2dced816e56beff0d38901aab0de290, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33489,1680720816595; forceNewPlan=false, retain=false 2023-04-05 18:54:43,126 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=c2dced816e56beff0d38901aab0de290, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:43,126 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720883126"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720883126"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720883126"}]},"ts":"1680720883126"} 2023-04-05 18:54:43,127 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:54:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 18:54:43,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:43,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c2dced816e56beff0d38901aab0de290, NAME => 'TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:54:43,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:43,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:54:43,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:43,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:43,283 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:43,285 DEBUG [StoreOpener-c2dced816e56beff0d38901aab0de290-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A 2023-04-05 18:54:43,285 DEBUG [StoreOpener-c2dced816e56beff0d38901aab0de290-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A 2023-04-05 18:54:43,285 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:54:43,285 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2dced816e56beff0d38901aab0de290 columnFamilyName A 2023-04-05 18:54:43,286 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] regionserver.HStore(310): Store=c2dced816e56beff0d38901aab0de290/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:54:43,286 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:43,287 DEBUG [StoreOpener-c2dced816e56beff0d38901aab0de290-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B 2023-04-05 18:54:43,287 DEBUG [StoreOpener-c2dced816e56beff0d38901aab0de290-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B 2023-04-05 18:54:43,287 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:54:43,287 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2dced816e56beff0d38901aab0de290 columnFamilyName B 2023-04-05 18:54:43,287 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] regionserver.HStore(310): Store=c2dced816e56beff0d38901aab0de290/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:54:43,288 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:43,288 DEBUG [StoreOpener-c2dced816e56beff0d38901aab0de290-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C 2023-04-05 18:54:43,289 DEBUG [StoreOpener-c2dced816e56beff0d38901aab0de290-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C 2023-04-05 18:54:43,289 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:54:43,289 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2dced816e56beff0d38901aab0de290 columnFamilyName C 2023-04-05 18:54:43,289 INFO [StoreOpener-c2dced816e56beff0d38901aab0de290-1] regionserver.HStore(310): Store=c2dced816e56beff0d38901aab0de290/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:54:43,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:43,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:43,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:43,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:54:43,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:43,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 18:54:43,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c2dced816e56beff0d38901aab0de290; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64213288, jitterRate=-0.04314744472503662}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:54:43,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:43,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., pid=30, masterSystemTime=1680720883278 2023-04-05 18:54:43,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:43,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:43,296 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=c2dced816e56beff0d38901aab0de290, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:43,296 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720883296"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680720883296"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680720883296"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680720883296"}]},"ts":"1680720883296"} 2023-04-05 18:54:43,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-05 18:54:43,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 in 170 msec 2023-04-05 18:54:43,300 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-05 18:54:43,300 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2dced816e56beff0d38901aab0de290, ASSIGN in 325 msec 2023-04-05 18:54:43,300 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 18:54:43,301 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720883300"}]},"ts":"1680720883300"} 2023-04-05 18:54:43,301 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 18:54:43,303 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 18:54:43,305 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1620 sec 2023-04-05 18:54:44,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 18:54:44,249 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-05 18:54:44,250 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,254 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,256 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,257 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34472, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,258 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 18:54:44,260 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44796, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 18:54:44,261 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,265 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,265 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,276 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,277 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,287 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,288 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x7810a231 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,295 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104dbf4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,296 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x08cd7cb0 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,301 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@753156b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,301 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,306 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,306 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,310 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,310 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,314 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,315 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x2de064cb to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,318 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f8ff99f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,318 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x54a39be8 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:54:44,322 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a4b7b8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:54:44,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:44,331 DEBUG [hconnection-0x44867287-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,332 DEBUG [hconnection-0x72d60ba3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,332 DEBUG [hconnection-0x195479aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,332 DEBUG [hconnection-0xebdadfe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,333 DEBUG [hconnection-0x783449dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,334 DEBUG [hconnection-0x2609cc90-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,334 DEBUG [hconnection-0x194fa333-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,335 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34484, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,335 DEBUG [hconnection-0x623945af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,335 DEBUG [hconnection-0x4e8f614c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,335 DEBUG [hconnection-0x3457eea8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:54:44,335 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34500, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,336 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34518, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,339 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34522, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,339 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,339 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34510, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,341 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34536, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,345 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34532, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,345 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34570, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,345 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34556, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:54:44,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:44,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:44,350 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:44,351 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:44,351 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:44,351 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:44,353 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,353 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,354 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:44,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:44,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,355 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,355 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,355 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:44,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:44,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:44,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:44,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:44,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:44,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:44,356 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:44,356 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:44,357 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 18:54:44,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:44,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:44,358 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:44,358 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:44,358 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:44,358 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:44,358 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:44,358 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:44,377 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/6e753e9f73c948fc8a4e5c8fcbd8504e 2023-04-05 18:54:44,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720884358, max timestamp=1680720884379Num uniques 859; 2023-04-05 18:54:44,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720884358, max timestamp=1680720884378Num uniques 859; 2023-04-05 18:54:44,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:44,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:44,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720884358, max timestamp=1680720884378Num uniques 859; 2023-04-05 18:54:44,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:44,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720884394, max timestamp=1680720884399Num uniques 1018; 2023-04-05 18:54:44,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720884394, max timestamp=1680720884399Num uniques 1018; 2023-04-05 18:54:44,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 18:54:44,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 18:54:44,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720884395, max timestamp=1680720884399Num uniques 1018; 2023-04-05 18:54:44,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 18:54:44,422 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/4784481d7c1342e78a3b953e40f351b9 2023-04-05 18:54:44,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720944417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720944417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720944421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720944422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720944423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,462 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/1c28af11683e498ab83f8a4a6630ec43 2023-04-05 18:54:44,466 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/6e753e9f73c948fc8a4e5c8fcbd8504e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/6e753e9f73c948fc8a4e5c8fcbd8504e 2023-04-05 18:54:44,472 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/6e753e9f73c948fc8a4e5c8fcbd8504e, entries=150, sequenceid=11, filesize=11.6 K 2023-04-05 18:54:44,473 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/4784481d7c1342e78a3b953e40f351b9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/4784481d7c1342e78a3b953e40f351b9 2023-04-05 18:54:44,480 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/4784481d7c1342e78a3b953e40f351b9, entries=150, sequenceid=11, filesize=11.6 K 2023-04-05 18:54:44,481 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/1c28af11683e498ab83f8a4a6630ec43 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1c28af11683e498ab83f8a4a6630ec43 2023-04-05 18:54:44,484 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1c28af11683e498ab83f8a4a6630ec43, entries=150, sequenceid=11, filesize=11.6 K 2023-04-05 18:54:44,485 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for c2dced816e56beff0d38901aab0de290 in 128ms, sequenceid=11, compaction requested=false 2023-04-05 18:54:44,485 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:44,485 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:44,485 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:44,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:44,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:44,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:44,487 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,487 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:44,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453c585c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:44,490 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:44,490 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,492 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,492 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,492 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:44,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:44,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:44,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:44,495 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:44,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:44,498 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:44,498 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:44,498 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:44,498 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:44,498 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:44,500 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,500 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:44,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,500 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:44,501 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,506 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,509 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:44,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:44,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,510 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:44,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:44,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:44,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:44,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:44,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:44,510 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:44,511 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:44,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:44,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:44,528 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=138.65 KB heapSize=394.30 KB 2023-04-05 18:54:44,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:44,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:44,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:44,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:44,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:44,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:44,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720944530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720944531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720944531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720944532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720944535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,611 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:44,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:44,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:44,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:44,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:44,617 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:44,617 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:44,617 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:44,617 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:44,618 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,618 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,619 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:44,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:44,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:44,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:44,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:44,621 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:44,621 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:44,621 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:44,622 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:44,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:44,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:44,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:44,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:44,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,624 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:44,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fcdf10a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:44,626 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:44,626 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:44,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:44,629 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,629 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:44,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,629 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:44,629 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:44,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:44,632 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:44,632 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:44,632 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:44,632 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:44,633 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:44,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:44,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,635 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,637 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720944636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720944636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720944636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720944636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720944638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,640 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:44,640 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:44,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:44,640 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,640 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:44,640 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,641 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,641 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,641 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,641 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:44,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:44,641 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:44,641 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:44,641 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:44,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:44,741 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:44,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:44,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:44,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:44,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:44,746 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:44,746 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:44,746 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:44,746 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:44,746 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,747 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,748 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:44,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:44,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:44,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:44,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:44,750 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:44,750 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:44,750 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:44,750 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:44,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,752 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,752 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:44,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456898bc[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:44,754 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:44,754 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:44,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:44,755 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,755 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:44,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:44,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:44,760 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:44,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:44,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:44,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:44,761 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:44,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,763 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,764 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:44,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:44,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,770 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:44,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:44,770 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:44,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:44,770 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:44,770 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,770 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,770 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,770 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:44,770 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,770 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:44,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:44,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720944840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720944841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720944841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720944841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:44,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720944841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,870 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:44,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:44,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:44,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:44,874 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:44,874 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:44,874 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:44,874 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:44,874 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,874 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,875 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:44,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:44,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:44,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:44,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:44,877 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:44,877 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:44,877 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:44,877 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:44,877 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:44,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:44,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:44,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:44,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:44,877 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:44,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:44,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:44,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:44,879 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:44,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ff8b7c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:44,882 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:44,882 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,884 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:44,884 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,884 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:44,885 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:44,886 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,886 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:44,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:44,888 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:44,888 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:44,888 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:44,888 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:44,888 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:44,890 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,890 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,890 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:44,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:44,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:44,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:44,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:44,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:44,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:44,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:44,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:44,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:44,895 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:44,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:44,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:44,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:44,895 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:44,896 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:44,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/8d22f68e40054443a61e9ddf060dc1cc 2023-04-05 18:54:44,996 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:44,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:44,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:44,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:44,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:44,999 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:44,999 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:44,999 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:44,999 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:45,000 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,000 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,002 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:45,002 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:45,002 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,002 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:45,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:45,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:45,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:45,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:45,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:45,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:45,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:45,003 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:45,003 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:45,005 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:45,005 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:45,005 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:45,005 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:45,005 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,005 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:45,005 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:45,006 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,006 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,007 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,007 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:45,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3726cf87[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:45,008 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:45,008 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,009 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:45,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:45,011 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:45,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:45,013 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:45,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:45,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:45,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:45,013 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:45,015 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,015 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,015 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:45,016 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,017 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,019 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,019 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,019 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,019 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:45,019 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,019 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:45,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:45,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:45,020 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,020 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,020 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:45,019 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:45,020 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:45,020 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:45,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:45,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:45,120 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:45,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:45,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:45,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:45,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:45,125 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:45,125 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:45,125 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:45,125 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:45,125 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,125 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,126 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,126 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:45,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:45,127 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,127 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:45,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:45,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:45,128 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:45,129 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:45,129 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:45,129 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:45,130 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,130 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,131 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,131 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f32dc8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:45,132 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:45,132 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,134 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:45,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:45,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:45,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:45,136 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:45,138 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:45,138 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:45,138 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:45,138 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:45,139 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:45,140 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,140 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,140 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:45,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:45,141 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,142 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:45,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:45,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:45,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:45,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,144 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:45,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:45,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:45,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:45,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,145 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:45,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720945143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,145 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:45,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:45,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720945143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720945144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720945144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720945146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,245 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:45,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:45,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:45,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:45,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:45,249 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:45,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:45,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:45,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:45,250 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,250 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,255 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:45,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:45,255 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:45,255 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:45,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:45,257 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:45,257 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:45,257 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:45,268 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:45,268 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:45,268 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:45,268 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:45,270 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:45,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26dfecf2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:45,272 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:45,272 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:45,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:45,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:45,274 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,274 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:45,276 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,276 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:45,276 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,276 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:45,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:45,278 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:45,278 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:45,278 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:45,278 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:45,279 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:45,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:45,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:45,281 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,282 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:45,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:45,285 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:45,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:45,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:45,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:45,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,285 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:45,286 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:45,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:45,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:45,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/03b51afbf0834e1daa3891f1877f36a4 2023-04-05 18:54:45,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/c553296f927f41dcbecb4f3e31c2542b 2023-04-05 18:54:45,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/8d22f68e40054443a61e9ddf060dc1cc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8d22f68e40054443a61e9ddf060dc1cc 2023-04-05 18:54:45,366 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8d22f68e40054443a61e9ddf060dc1cc, entries=200, sequenceid=65, filesize=13.9 K 2023-04-05 18:54:45,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/03b51afbf0834e1daa3891f1877f36a4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/03b51afbf0834e1daa3891f1877f36a4 2023-04-05 18:54:45,371 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/03b51afbf0834e1daa3891f1877f36a4, entries=150, sequenceid=65, filesize=11.6 K 2023-04-05 18:54:45,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/c553296f927f41dcbecb4f3e31c2542b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/c553296f927f41dcbecb4f3e31c2542b 2023-04-05 18:54:45,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/c553296f927f41dcbecb4f3e31c2542b, entries=150, sequenceid=65, filesize=11.6 K 2023-04-05 18:54:45,378 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~394.25 KB/403712, currentSize=40.25 KB/41220 for c2dced816e56beff0d38901aab0de290 in 850ms, sequenceid=65, compaction requested=false 2023-04-05 18:54:45,378 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:45,386 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:45,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:45,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:45,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:45,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:45,389 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:45,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:45,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:45,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:45,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,391 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:45,391 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:45,391 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,391 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:45,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:45,393 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:45,393 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:45,393 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 18:54:45,393 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:45,393 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:45,393 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:45,393 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:45,393 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:45,393 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:45,404 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/1bfcec3172ff4ddaa542f535415fceb2 2023-04-05 18:54:45,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:45,650 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:45,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720885658, max timestamp=1680720885663Num uniques 859; 2023-04-05 18:54:45,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:45,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720945661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720885658, max timestamp=1680720885663Num uniques 859; 2023-04-05 18:54:45,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720885658, max timestamp=1680720885663Num uniques 859; 2023-04-05 18:54:45,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:45,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:45,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720945678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720945678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720885675, max timestamp=1680720885680Num uniques 1018; 2023-04-05 18:54:45,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 18:54:45,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720885675, max timestamp=1680720885680Num uniques 1018; 2023-04-05 18:54:45,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680720885675, max timestamp=1680720885680Num uniques 1018; 2023-04-05 18:54:45,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 18:54:45,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 18:54:45,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720945710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720945711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720945766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720945783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720945784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,816 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/2b34d0c5d65b4ee4a882372f58477c47 2023-04-05 18:54:45,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720945813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720945814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,833 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/7b8f12fb65c44ffd8b38738352e19abb 2023-04-05 18:54:45,837 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/1bfcec3172ff4ddaa542f535415fceb2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/1bfcec3172ff4ddaa542f535415fceb2 2023-04-05 18:54:45,841 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/1bfcec3172ff4ddaa542f535415fceb2, entries=150, sequenceid=74, filesize=11.6 K 2023-04-05 18:54:45,842 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/2b34d0c5d65b4ee4a882372f58477c47 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b34d0c5d65b4ee4a882372f58477c47 2023-04-05 18:54:45,846 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b34d0c5d65b4ee4a882372f58477c47, entries=150, sequenceid=74, filesize=11.6 K 2023-04-05 18:54:45,846 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/7b8f12fb65c44ffd8b38738352e19abb as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7b8f12fb65c44ffd8b38738352e19abb 2023-04-05 18:54:45,850 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7b8f12fb65c44ffd8b38738352e19abb, entries=150, sequenceid=74, filesize=11.6 K 2023-04-05 18:54:45,851 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for c2dced816e56beff0d38901aab0de290 in 458ms, sequenceid=74, compaction requested=true 2023-04-05 18:54:45,851 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:45,851 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:45,851 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:45,851 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:45,851 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,851 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:45,852 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:45,854 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,854 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,854 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,854 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:45,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a97273e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:45,856 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:45,856 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,858 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:45,858 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,858 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:45,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:45,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:45,860 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:45,862 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:45,862 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:45,862 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:45,863 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:45,863 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:45,864 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,864 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,864 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:45,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:45,865 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,866 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:45,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:45,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:45,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:45,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:45,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,870 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:45,870 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:45,870 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:45,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:45,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:45,971 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:45,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:45,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:45,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=147.60 KB heapSize=420 KB 2023-04-05 18:54:45,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:45,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:45,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:45,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:45,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:45,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 18:54:45,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:45,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:45,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:45,975 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:45,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:45,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:45,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:45,976 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,976 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,977 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,977 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,978 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:45,977 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:45,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:45,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:45,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:45,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:45,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:45,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:45,979 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:45,979 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:45,979 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:45,979 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:45,981 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,981 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,981 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,981 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bcc931c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:45,983 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:45,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,985 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,985 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:45,985 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:45,985 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:45,985 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,985 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:45,987 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:45,987 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:45,987 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,987 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:45,990 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:45,990 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:45,990 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:45,990 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:45,990 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:45,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720945989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,992 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:45,992 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,993 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:45,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:45,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720945991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:45,994 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720945991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,995 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,997 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,997 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:45,997 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:45,997 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:45,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:45,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:45,997 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:45,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:45,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:45,998 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:45,998 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:45,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:45,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:45,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720946018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720946019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720946094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720946095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,098 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:46,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:46,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:46,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:46,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:46,101 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:46,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:46,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:46,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:46,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:46,103 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:46,103 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,104 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:46,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:46,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:46,105 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:46,105 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:46,105 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:46,105 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:46,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,108 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:46,108 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,108 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:46,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:46,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@756f7ac0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:46,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:46,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,113 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:46,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:46,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:46,114 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,114 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:46,115 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,115 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:46,115 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:46,115 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:46,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:46,118 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:46,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:46,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:46,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:46,118 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:46,119 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,119 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,120 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:46,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:46,120 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,121 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,123 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,123 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:46,123 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:46,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:46,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:46,123 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,123 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:46,124 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:46,124 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,124 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,124 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:46,124 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:46,124 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:46,224 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:46,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:46,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:46,228 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:46,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:46,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:46,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:46,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,229 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,231 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:46,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:46,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:46,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:46,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:46,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:46,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:46,235 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:46,235 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:46,235 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:46,235 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,236 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:46,236 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:46,238 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,238 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:46,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:46,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c0ea0e3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:46,245 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:46,245 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:46,248 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:46,249 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:46,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:46,253 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:46,253 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:46,253 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:46,253 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:46,253 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:46,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:46,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:46,258 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,260 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,263 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:46,263 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:46,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:46,263 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,264 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:46,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:46,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:46,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:46,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:46,264 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:46,264 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:46,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720946294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720946297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720946298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720946321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720946322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,364 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:46,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:46,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:46,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:46,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:46,368 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:46,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:46,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:46,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:46,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:46,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:46,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:46,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:46,374 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:46,374 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:46,374 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:46,374 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:46,375 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:46,376 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,376 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd3dc8b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:46,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:46,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,379 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:46,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:46,379 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:46,380 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,380 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:46,382 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:46,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/43101259b5da4beca76d0baccabf003b 2023-04-05 18:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:46,393 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:46,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:46,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:46,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:46,393 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:46,395 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,395 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,395 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:46,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,397 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,400 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,400 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,400 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,400 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:46,400 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:46,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:46,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:46,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:46,400 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:46,400 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:46,401 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,401 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:46,401 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:46,401 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:46,401 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:46,403 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/c5968215bfd44a87ac1ec0db1e21978d 2023-04-05 18:54:46,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/f10f6bf9aa5840949b43f324f0e66eb6 2023-04-05 18:54:46,460 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/43101259b5da4beca76d0baccabf003b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/43101259b5da4beca76d0baccabf003b 2023-04-05 18:54:46,468 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/43101259b5da4beca76d0baccabf003b, entries=150, sequenceid=129, filesize=11.7 K 2023-04-05 18:54:46,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/c5968215bfd44a87ac1ec0db1e21978d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c5968215bfd44a87ac1ec0db1e21978d 2023-04-05 18:54:46,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c5968215bfd44a87ac1ec0db1e21978d, entries=150, sequenceid=129, filesize=11.7 K 2023-04-05 18:54:46,478 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/f10f6bf9aa5840949b43f324f0e66eb6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/f10f6bf9aa5840949b43f324f0e66eb6 2023-04-05 18:54:46,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/f10f6bf9aa5840949b43f324f0e66eb6, entries=150, sequenceid=129, filesize=11.7 K 2023-04-05 18:54:46,483 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~419.95 KB/430032, currentSize=33.54 KB/34350 for c2dced816e56beff0d38901aab0de290 in 511ms, sequenceid=129, compaction requested=true 2023-04-05 18:54:46,483 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:46,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:46,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:54:46,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-05 18:54:46,484 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:54:46,484 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:54:46,485 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:54:46,486 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/C is initiating minor compaction (all files) 2023-04-05 18:54:46,486 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/C in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:46,486 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1c28af11683e498ab83f8a4a6630ec43, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/c553296f927f41dcbecb4f3e31c2542b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7b8f12fb65c44ffd8b38738352e19abb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/f10f6bf9aa5840949b43f324f0e66eb6] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=46.5 K 2023-04-05 18:54:46,487 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1c28af11683e498ab83f8a4a6630ec43, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680720884355 2023-04-05 18:54:46,487 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49934 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:54:46,487 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/A is initiating minor compaction (all files) 2023-04-05 18:54:46,487 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/A in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:46,487 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/6e753e9f73c948fc8a4e5c8fcbd8504e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8d22f68e40054443a61e9ddf060dc1cc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/1bfcec3172ff4ddaa542f535415fceb2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/43101259b5da4beca76d0baccabf003b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=48.8 K 2023-04-05 18:54:46,488 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting c553296f927f41dcbecb4f3e31c2542b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680720884416 2023-04-05 18:54:46,488 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6e753e9f73c948fc8a4e5c8fcbd8504e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680720884355 2023-04-05 18:54:46,488 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 7b8f12fb65c44ffd8b38738352e19abb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680720884529 2023-04-05 18:54:46,489 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 8d22f68e40054443a61e9ddf060dc1cc, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680720884416 2023-04-05 18:54:46,489 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting f10f6bf9aa5840949b43f324f0e66eb6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680720885698 2023-04-05 18:54:46,489 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 1bfcec3172ff4ddaa542f535415fceb2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680720884529 2023-04-05 18:54:46,490 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 43101259b5da4beca76d0baccabf003b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680720885698 2023-04-05 18:54:46,498 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#C#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:46,501 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:46,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:46,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:46,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:46,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:46,515 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:46,515 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:46,515 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:46,515 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:46,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,517 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#A#compaction#210 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:46,518 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:46,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:46,518 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,519 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,519 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:46,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:46,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:46,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:46,546 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:46,546 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:46,546 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:46,552 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:46,552 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:46,553 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 18:54:46,553 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:46,553 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:46,553 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:46,554 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:46,554 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:46,554 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:46,567 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/f7403edb09034f1c9819e4e29720d412 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/f7403edb09034f1c9819e4e29720d412 2023-04-05 18:54:46,578 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2dced816e56beff0d38901aab0de290/A of c2dced816e56beff0d38901aab0de290 into f7403edb09034f1c9819e4e29720d412(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:46,578 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:46,578 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/A, priority=12, startTime=1680720886483; duration=0sec 2023-04-05 18:54:46,578 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:46,578 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:54:46,580 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:54:46,580 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/B is initiating minor compaction (all files) 2023-04-05 18:54:46,580 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/B in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:46,581 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/4784481d7c1342e78a3b953e40f351b9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/03b51afbf0834e1daa3891f1877f36a4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b34d0c5d65b4ee4a882372f58477c47, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c5968215bfd44a87ac1ec0db1e21978d] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=46.5 K 2023-04-05 18:54:46,581 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 4784481d7c1342e78a3b953e40f351b9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680720884355 2023-04-05 18:54:46,582 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 03b51afbf0834e1daa3891f1877f36a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680720884416 2023-04-05 18:54:46,582 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 2b34d0c5d65b4ee4a882372f58477c47, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680720884529 2023-04-05 18:54:46,583 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c5968215bfd44a87ac1ec0db1e21978d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680720885698 2023-04-05 18:54:46,592 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#B#compaction#212 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:46,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:46,606 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:46,618 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/0c98276ef3d94c84a7d04d77cf7de601 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/0c98276ef3d94c84a7d04d77cf7de601 2023-04-05 18:54:46,624 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2dced816e56beff0d38901aab0de290/B of c2dced816e56beff0d38901aab0de290 into 0c98276ef3d94c84a7d04d77cf7de601(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:46,624 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:46,624 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/B, priority=12, startTime=1680720886483; duration=0sec 2023-04-05 18:54:46,624 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:46,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720886620, max timestamp=1680720886641Num uniques 859; 2023-04-05 18:54:46,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:46,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720886620, max timestamp=1680720886641Num uniques 859; 2023-04-05 18:54:46,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 18:54:46,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720886620, max timestamp=1680720886641Num uniques 859; 2023-04-05 18:54:46,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 18:54:46,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720886675, max timestamp=1680720886684Num uniques 1018; 2023-04-05 18:54:46,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 18:54:46,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720886675, max timestamp=1680720886684Num uniques 1009; 2023-04-05 18:54:46,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-05 18:54:46,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720886675, max timestamp=1680720886686Num uniques 1009; 2023-04-05 18:54:46,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-05 18:54:46,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680720886705, max timestamp=1680720886717Num uniques 1027; 2023-04-05 18:54:46,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-05 18:54:46,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680720886705, max timestamp=1680720886717Num uniques 1027; 2023-04-05 18:54:46,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680720886705, max timestamp=1680720886717Num uniques 1027; 2023-04-05 18:54:46,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-05 18:54:46,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-05 18:54:46,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680720886735, max timestamp=1680720886754Num uniques 1036; 2023-04-05 18:54:46,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-05 18:54:46,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680720886735, max timestamp=1680720886754Num uniques 1036; 2023-04-05 18:54:46,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-05 18:54:46,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680720886735, max timestamp=1680720886754Num uniques 1036; 2023-04-05 18:54:46,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-05 18:54:46,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680720886778, max timestamp=1680720886790Num uniques 1095; 2023-04-05 18:54:46,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-05 18:54:46,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680720886778, max timestamp=1680720886790Num uniques 1095; 2023-04-05 18:54:46,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-05 18:54:46,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680720886778, max timestamp=1680720886790Num uniques 1095; 2023-04-05 18:54:46,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-05 18:54:46,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720886812, max timestamp=1680720886822Num uniques 1054; 2023-04-05 18:54:46,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 18:54:46,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720886812, max timestamp=1680720886822Num uniques 1054; 2023-04-05 18:54:46,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 18:54:46,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720886812, max timestamp=1680720886822Num uniques 1054; 2023-04-05 18:54:46,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 18:54:46,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720946841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680720886840, max timestamp=1680720886845Num uniques 1013; 2023-04-05 18:54:46,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-05 18:54:46,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680720886840, max timestamp=1680720886845Num uniques 1013; 2023-04-05 18:54:46,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-05 18:54:46,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680720886840, max timestamp=1680720886841Num uniques 1013; 2023-04-05 18:54:46,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-05 18:54:46,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720946896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680720886892, max timestamp=1680720886898Num uniques 1022; 2023-04-05 18:54:46,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-05 18:54:46,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680720886892, max timestamp=1680720886898Num uniques 1022; 2023-04-05 18:54:46,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-05 18:54:46,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680720886892, max timestamp=1680720886898Num uniques 1022; 2023-04-05 18:54:46,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-05 18:54:46,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720946931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680720886920, max timestamp=1680720886931Num uniques 1081; 2023-04-05 18:54:46,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680720886920, max timestamp=1680720886931Num uniques 1081; 2023-04-05 18:54:46,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-05 18:54:46,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680720886920, max timestamp=1680720886931Num uniques 1081; 2023-04-05 18:54:46,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-05 18:54:46,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-05 18:54:46,948 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/db6f707b14a94da0b32dc8b9f2ed2ef1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/db6f707b14a94da0b32dc8b9f2ed2ef1 2023-04-05 18:54:46,963 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2dced816e56beff0d38901aab0de290/C of c2dced816e56beff0d38901aab0de290 into db6f707b14a94da0b32dc8b9f2ed2ef1(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:46,963 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:46,963 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/C, priority=12, startTime=1680720886483; duration=0sec 2023-04-05 18:54:46,963 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:46,966 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/9bf6408917344eb282b18d50df85d6af 2023-04-05 18:54:46,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720946969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720946972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:46,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:46,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720946975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720947001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720947036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720947072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720947075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720947179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720947204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720947239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720947275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720947277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,380 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/2337ad528a1143509adf5c626ebd3ce4 2023-04-05 18:54:47,389 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 18:54:47,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720947483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720947507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720947544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720947578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:47,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720947580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,791 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/dc6b0ed18b4b434cb937d02ea664b4aa 2023-04-05 18:54:47,795 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/9bf6408917344eb282b18d50df85d6af as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9bf6408917344eb282b18d50df85d6af 2023-04-05 18:54:47,799 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9bf6408917344eb282b18d50df85d6af, entries=150, sequenceid=137, filesize=11.8 K 2023-04-05 18:54:47,800 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/2337ad528a1143509adf5c626ebd3ce4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2337ad528a1143509adf5c626ebd3ce4 2023-04-05 18:54:47,804 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2337ad528a1143509adf5c626ebd3ce4, entries=150, sequenceid=137, filesize=11.8 K 2023-04-05 18:54:47,804 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/dc6b0ed18b4b434cb937d02ea664b4aa as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/dc6b0ed18b4b434cb937d02ea664b4aa 2023-04-05 18:54:47,808 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/dc6b0ed18b4b434cb937d02ea664b4aa, entries=150, sequenceid=137, filesize=11.8 K 2023-04-05 18:54:47,809 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for c2dced816e56beff0d38901aab0de290 in 1257ms, sequenceid=137, compaction requested=false 2023-04-05 18:54:47,809 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:47,809 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:47,809 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:47,809 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:47,809 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,809 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:47,809 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:47,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:47,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:47,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7028f01e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:47,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:47,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:47,814 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:47,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:47,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:47,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:47,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:47,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:47,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:47,823 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:47,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:47,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:47,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:47,823 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:47,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:47,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:47,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:47,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:47,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:47,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:47,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:47,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:47,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:47,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:47,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:47,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:47,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:47,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:47,829 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:47,829 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:47,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:47,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:47,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:47,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:47,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:47,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:47,929 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:47,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:47,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:47,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:47,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:47,934 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:47,934 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:47,934 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:47,934 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:47,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:47,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:47,936 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,937 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:47,937 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:47,937 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:47,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:47,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:47,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:47,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:47,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:47,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:47,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:47,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:47,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:47,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:47,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:47,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:47,939 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:47,939 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:47,939 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=147.60 KB heapSize=413.39 KB 2023-04-05 18:54:47,939 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:47,939 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:47,940 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:47,940 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:47,940 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:47,940 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:47,960 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/267d9890552c442aa78a4f5791a25098 2023-04-05 18:54:47,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:47,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:48,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:48,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720948001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:48,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34570 deadline: 1680720948011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:48,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720948048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:48,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720948084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:48,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34500 deadline: 1680720948087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:48,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720948105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:48,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720948309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,374 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/c897ffa0d673456a846989e056d87184 2023-04-05 18:54:48,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:48,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720948613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,785 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/6dd137d21b964616bd50013d84b41274 2023-04-05 18:54:48,790 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/267d9890552c442aa78a4f5791a25098 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/267d9890552c442aa78a4f5791a25098 2023-04-05 18:54:48,795 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/267d9890552c442aa78a4f5791a25098, entries=150, sequenceid=314, filesize=11.9 K 2023-04-05 18:54:48,796 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/c897ffa0d673456a846989e056d87184 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c897ffa0d673456a846989e056d87184 2023-04-05 18:54:48,803 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c897ffa0d673456a846989e056d87184, entries=150, sequenceid=314, filesize=11.9 K 2023-04-05 18:54:48,804 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/6dd137d21b964616bd50013d84b41274 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/6dd137d21b964616bd50013d84b41274 2023-04-05 18:54:48,818 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/6dd137d21b964616bd50013d84b41274, entries=150, sequenceid=314, filesize=11.9 K 2023-04-05 18:54:48,818 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~413.34 KB/423264, currentSize=33.54 KB/34350 for c2dced816e56beff0d38901aab0de290 in 879ms, sequenceid=314, compaction requested=true 2023-04-05 18:54:48,819 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:48,819 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:48,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:48,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:48,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:48,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:48,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,821 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:48,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:48,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:48,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:48,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e1f2da1[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:48,824 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:48,824 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:48,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:48,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:48,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:48,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:48,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:48,827 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:48,827 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:48,827 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:48,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:48,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:48,828 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:48,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:48,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:48,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:48,828 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:48,831 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:48,831 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:48,831 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:48,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:48,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:48,833 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,834 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,838 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,838 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:48,838 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:48,838 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:48,838 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,838 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:48,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:48,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:48,838 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:48,838 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:48,838 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:48,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:48,838 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:48,838 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:48,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:48,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:48,938 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:48,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:48,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:48,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:48,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:48,942 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:48,943 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:48,943 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:48,943 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:48,943 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:48,943 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:48,944 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:48,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:48,944 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:48,945 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:48,945 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:48,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:48,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:48,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:48,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:48,946 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:48,946 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:48,946 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:48,946 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:48,946 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:48,946 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:48,946 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:48,946 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 18:54:48,946 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:48,946 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:48,946 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:48,946 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:48,946 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:48,947 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:48,964 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/c98539fd2de4418faaa0b10dbd735df5 2023-04-05 18:54:48,978 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/71eef36d4e164de881948ca48dedfe48 2023-04-05 18:54:48,989 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/16cb460b9a1d40d3865f09df04f9865e 2023-04-05 18:54:48,993 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/c98539fd2de4418faaa0b10dbd735df5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c98539fd2de4418faaa0b10dbd735df5 2023-04-05 18:54:48,997 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c98539fd2de4418faaa0b10dbd735df5, entries=150, sequenceid=322, filesize=11.9 K 2023-04-05 18:54:48,997 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/71eef36d4e164de881948ca48dedfe48 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/71eef36d4e164de881948ca48dedfe48 2023-04-05 18:54:49,001 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/71eef36d4e164de881948ca48dedfe48, entries=150, sequenceid=322, filesize=11.9 K 2023-04-05 18:54:49,002 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/16cb460b9a1d40d3865f09df04f9865e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/16cb460b9a1d40d3865f09df04f9865e 2023-04-05 18:54:49,005 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/16cb460b9a1d40d3865f09df04f9865e, entries=150, sequenceid=322, filesize=11.9 K 2023-04-05 18:54:49,007 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for c2dced816e56beff0d38901aab0de290 in 61ms, sequenceid=322, compaction requested=true 2023-04-05 18:54:49,007 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:49,007 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,007 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:49,007 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:49,007 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,007 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:49,007 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,009 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,010 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,010 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:49,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@274f9339[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:49,012 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:49,012 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:49,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:49,013 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:49,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:49,016 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:49,016 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,016 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:49,016 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:49,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:49,019 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:49,019 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:49,019 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:49,019 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:49,019 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:49,021 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,021 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,021 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,023 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,024 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:49,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:49,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,027 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,027 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:49,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,027 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:49,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:49,039 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:54:49,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:49,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:49,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:49,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:49,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:49,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720889077, max timestamp=1680720889087Num uniques 859; 2023-04-05 18:54:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720889077, max timestamp=1680720889087Num uniques 859; 2023-04-05 18:54:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720889077, max timestamp=1680720889087Num uniques 859; 2023-04-05 18:54:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:49,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720949107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720949108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720949109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720949112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720949116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,127 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:49,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:49,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:49,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:49,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:49,131 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:49,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:49,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:49,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,132 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,132 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,133 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,133 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:49,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:49,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:49,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:49,134 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,134 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:49,134 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:49,134 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:49,137 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b924df[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:49,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:49,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,140 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:49,141 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,141 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:49,142 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:49,145 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:49,145 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:49,145 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:49,145 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:49,145 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:49,147 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,148 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,150 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,155 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:49,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:49,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:49,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,156 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:49,156 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:49,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720949213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720949213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720949213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720949217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,256 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:49,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:49,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:49,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:49,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:49,259 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:49,259 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,260 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:49,260 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:49,260 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,260 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,261 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,261 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,262 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,262 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:49,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:49,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:49,265 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a69016[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:49,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:49,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,268 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:49,268 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,268 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:49,270 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,271 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:49,271 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,271 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:49,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:49,273 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:49,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:49,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:49,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:49,273 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:49,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,275 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,276 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:49,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:49,280 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:49,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,280 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:49,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,280 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:49,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,380 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:49,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:49,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:49,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:49,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:49,384 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:49,384 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,384 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:49,384 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:49,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,386 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:49,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:49,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:49,388 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:49,388 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,388 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:49,389 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:49,389 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:49,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:49,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:49,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,395 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,395 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:49,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:49,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b1f7e9c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:49,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:49,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:49,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,398 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:49,398 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,398 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,400 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:49,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:49,403 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:49,403 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:49,403 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:49,404 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:49,404 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:49,406 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,407 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,407 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,409 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,411 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,415 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:49,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,416 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:49,416 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:49,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720949417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720949417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720949418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720949421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/9dde5f43e93844929895ae586e33f0f6 2023-04-05 18:54:49,516 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:49,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:49,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:49,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:49,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:49,519 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:49,519 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,519 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:49,519 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:49,520 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,520 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,521 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,521 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,522 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,522 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:49,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:49,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:49,523 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,523 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:49,523 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:49,523 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:49,525 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,525 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,525 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,525 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:49,527 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:49,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5653cda6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:49,527 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,528 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:49,528 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,528 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:49,534 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:49,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:49,537 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:49,537 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:49,537 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:49,537 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:49,537 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:49,539 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,539 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,539 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,540 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:49,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:49,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,545 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,546 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,546 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,546 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:49,546 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,546 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:49,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,646 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:49,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:49,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:49,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:49,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:49,649 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:49,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:49,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:49,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,652 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,653 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,653 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:49,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:49,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:49,654 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,654 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:49,654 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:49,654 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:49,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,656 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,656 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,656 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dde5a0e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:49,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:49,658 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,659 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,659 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:49,659 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:49,659 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:49,659 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,659 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:49,660 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:49,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:49,663 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:49,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:49,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:49,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:49,663 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:49,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,665 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,667 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:49,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:49,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:49,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,669 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:49,669 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:49,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720949720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720949720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720949721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:49,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720949727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,770 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:49,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:49,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:49,773 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:49,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:49,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:49,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,774 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,774 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,775 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,775 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:49,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:49,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:49,776 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,776 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:49,776 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:49,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:49,778 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:49,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226e08b6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:49,780 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:49,780 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,782 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:49,782 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,782 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:49,783 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:49,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:49,786 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:49,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:49,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:49,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:49,786 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:49,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,788 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,791 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:49,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:49,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,792 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,792 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:49,792 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:49,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:49,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/8745866b265f4703b44c7f00af26aeb2 2023-04-05 18:54:49,892 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:49,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:49,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:49,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:49,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:49,895 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:49,895 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,895 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:49,896 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:49,896 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,896 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,897 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,897 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,898 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,898 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:49,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:49,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:49,899 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,899 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:49,899 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:49,899 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:49,901 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,901 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,901 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,901 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:49,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@472d1acd[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:49,903 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:49,903 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,904 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,904 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:49,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:49,913 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:49,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:49,915 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:49,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:49,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:49,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:49,915 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:49,917 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,917 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:49,917 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:49,918 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,919 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,921 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,921 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,921 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:49,921 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:49,921 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:49,921 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,921 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:49,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:49,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:49,921 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:49,921 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:49,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:49,921 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:49,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:49,921 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:49,921 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:49,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:50,021 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:50,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:50,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:50,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:50,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:50,025 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:50,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:50,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:50,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:50,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:50,030 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:50,030 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,030 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:50,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:50,031 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:50,032 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:50,032 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:50,032 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:50,032 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:50,036 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:50,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f5bc173[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:50,038 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:50,038 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,039 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,039 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:50,039 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:50,039 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:50,039 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,039 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:50,040 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:50,043 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:50,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:50,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:50,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:50,043 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:50,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:50,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,046 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:50,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:50,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,048 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:50,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:50,050 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:50,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:50,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:50,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:50,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,051 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,051 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,051 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:50,051 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:50,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:50,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:50,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720950124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,151 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:50,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:50,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:50,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:50,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:50,154 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:50,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:50,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:50,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:50,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,155 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,156 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:50,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:50,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,156 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:50,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:50,158 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:50,158 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:50,158 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:50,158 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,160 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@702af4bc[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:50,161 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:50,161 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,163 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,163 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,163 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:50,165 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:50,167 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:50,167 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:50,167 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:50,167 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:50,167 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:50,169 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,169 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:50,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,169 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:50,170 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,171 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:50,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:50,173 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:50,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:50,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:50,173 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:50,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:50,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:50,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,173 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:50,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:50,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:50,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720950225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:50,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720950226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:50,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720950227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:50,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720950230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,274 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:50,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/67725fc7df7643fbaca5a313129de54c 2023-04-05 18:54:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:50,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:50,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:50,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:50,278 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:50,278 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:50,278 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:50,278 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:50,278 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,278 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/9dde5f43e93844929895ae586e33f0f6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9dde5f43e93844929895ae586e33f0f6 2023-04-05 18:54:50,281 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:50,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:50,281 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,282 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,282 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:50,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:50,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:50,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:50,283 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:50,283 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:50,284 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:50,284 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:50,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9dde5f43e93844929895ae586e33f0f6, entries=200, sequenceid=332, filesize=14.3 K 2023-04-05 18:54:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:50,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/8745866b265f4703b44c7f00af26aeb2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8745866b265f4703b44c7f00af26aeb2 2023-04-05 18:54:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d379b7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:50,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:50,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,291 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,291 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:50,291 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:50,291 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:50,291 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,291 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:50,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8745866b265f4703b44c7f00af26aeb2, entries=150, sequenceid=332, filesize=11.9 K 2023-04-05 18:54:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:50,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/67725fc7df7643fbaca5a313129de54c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/67725fc7df7643fbaca5a313129de54c 2023-04-05 18:54:50,296 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:50,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:50,299 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:50,299 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:50,299 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:50,299 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:50,300 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:50,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/67725fc7df7643fbaca5a313129de54c, entries=150, sequenceid=332, filesize=11.9 K 2023-04-05 18:54:50,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c2dced816e56beff0d38901aab0de290 in 1261ms, sequenceid=332, compaction requested=true 2023-04-05 18:54:50,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:50,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:50,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:50,301 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:54:50,301 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:54:50,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:50,302 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63108 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:54:50,302 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/A is initiating minor compaction (all files) 2023-04-05 18:54:50,302 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60668 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:54:50,302 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/A in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:50,302 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/B is initiating minor compaction (all files) 2023-04-05 18:54:50,302 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/f7403edb09034f1c9819e4e29720d412, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9bf6408917344eb282b18d50df85d6af, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/267d9890552c442aa78a4f5791a25098, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c98539fd2de4418faaa0b10dbd735df5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9dde5f43e93844929895ae586e33f0f6] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=61.6 K 2023-04-05 18:54:50,302 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/B in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:50,302 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/0c98276ef3d94c84a7d04d77cf7de601, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2337ad528a1143509adf5c626ebd3ce4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c897ffa0d673456a846989e056d87184, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/71eef36d4e164de881948ca48dedfe48, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8745866b265f4703b44c7f00af26aeb2] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=59.2 K 2023-04-05 18:54:50,303 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting f7403edb09034f1c9819e4e29720d412, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680720885698 2023-04-05 18:54:50,303 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0c98276ef3d94c84a7d04d77cf7de601, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680720885698 2023-04-05 18:54:50,303 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,303 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,303 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:50,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,303 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 9bf6408917344eb282b18d50df85d6af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680720885975 2023-04-05 18:54:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:50,303 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 2337ad528a1143509adf5c626ebd3ce4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680720885975 2023-04-05 18:54:50,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:50,305 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 267d9890552c442aa78a4f5791a25098, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680720886966 2023-04-05 18:54:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,305 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting c897ffa0d673456a846989e056d87184, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680720886966 2023-04-05 18:54:50,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,305 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c98539fd2de4418faaa0b10dbd735df5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680720887994 2023-04-05 18:54:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:50,306 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 71eef36d4e164de881948ca48dedfe48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680720887994 2023-04-05 18:54:50,306 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 9dde5f43e93844929895ae586e33f0f6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680720889023 2023-04-05 18:54:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,306 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 8745866b265f4703b44c7f00af26aeb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680720889023 2023-04-05 18:54:50,307 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:50,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:50,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:50,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:50,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:50,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:50,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:50,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:50,311 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:50,311 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:50,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:50,319 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#B#compaction#224 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:50,323 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#A#compaction#225 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:50,338 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/0a1ba0c8feff4740a25cb117e199e134 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/0a1ba0c8feff4740a25cb117e199e134 2023-04-05 18:54:50,339 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/2ad042cbc18f4f00b9ee5873cf8ffee2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/2ad042cbc18f4f00b9ee5873cf8ffee2 2023-04-05 18:54:50,344 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c2dced816e56beff0d38901aab0de290/B of c2dced816e56beff0d38901aab0de290 into 0a1ba0c8feff4740a25cb117e199e134(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:50,344 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:50,344 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/B, priority=11, startTime=1680720890301; duration=0sec 2023-04-05 18:54:50,344 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c2dced816e56beff0d38901aab0de290/A of c2dced816e56beff0d38901aab0de290 into 2ad042cbc18f4f00b9ee5873cf8ffee2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:50,344 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:50,344 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:50,344 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/A, priority=11, startTime=1680720890301; duration=0sec 2023-04-05 18:54:50,344 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:54:50,345 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:50,347 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60668 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:54:50,347 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/C is initiating minor compaction (all files) 2023-04-05 18:54:50,347 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/C in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:50,347 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/db6f707b14a94da0b32dc8b9f2ed2ef1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/dc6b0ed18b4b434cb937d02ea664b4aa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/6dd137d21b964616bd50013d84b41274, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/16cb460b9a1d40d3865f09df04f9865e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/67725fc7df7643fbaca5a313129de54c] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=59.2 K 2023-04-05 18:54:50,348 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting db6f707b14a94da0b32dc8b9f2ed2ef1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680720885698 2023-04-05 18:54:50,348 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting dc6b0ed18b4b434cb937d02ea664b4aa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680720885975 2023-04-05 18:54:50,349 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 6dd137d21b964616bd50013d84b41274, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680720886966 2023-04-05 18:54:50,349 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 16cb460b9a1d40d3865f09df04f9865e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680720887994 2023-04-05 18:54:50,349 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 67725fc7df7643fbaca5a313129de54c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680720889023 2023-04-05 18:54:50,358 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#C#compaction#226 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:50,411 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:50,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:50,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:50,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:50,414 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:50,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:50,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:50,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:50,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,415 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:50,415 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:50,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:50,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:50,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:50,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:50,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:50,417 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:50,417 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:50,417 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-05 18:54:50,417 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:50,417 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:50,417 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:50,418 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:50,418 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:50,418 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:50,424 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/c9807f17e20f4e38926d203787fcfbe6 2023-04-05 18:54:50,774 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/7f7a3fd1922d40328177114a304b95ed as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7f7a3fd1922d40328177114a304b95ed 2023-04-05 18:54:50,779 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c2dced816e56beff0d38901aab0de290/C of c2dced816e56beff0d38901aab0de290 into 7f7a3fd1922d40328177114a304b95ed(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:50,779 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:50,779 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/C, priority=11, startTime=1680720890301; duration=0sec 2023-04-05 18:54:50,779 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:50,836 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/c49c2efefa0b417bb8f5978306daddc5 2023-04-05 18:54:51,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:51,233 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:51,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720951240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720951240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720951242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720951242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,248 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/2fabf7552fb74a6a9828e5b2848742e2 2023-04-05 18:54:51,253 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/c9807f17e20f4e38926d203787fcfbe6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9807f17e20f4e38926d203787fcfbe6 2023-04-05 18:54:51,259 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9807f17e20f4e38926d203787fcfbe6, entries=150, sequenceid=371, filesize=11.9 K 2023-04-05 18:54:51,268 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/c49c2efefa0b417bb8f5978306daddc5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c49c2efefa0b417bb8f5978306daddc5 2023-04-05 18:54:51,273 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c49c2efefa0b417bb8f5978306daddc5, entries=150, sequenceid=371, filesize=11.9 K 2023-04-05 18:54:51,274 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/2fabf7552fb74a6a9828e5b2848742e2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/2fabf7552fb74a6a9828e5b2848742e2 2023-04-05 18:54:51,281 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/2fabf7552fb74a6a9828e5b2848742e2, entries=150, sequenceid=371, filesize=11.9 K 2023-04-05 18:54:51,282 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for c2dced816e56beff0d38901aab0de290 in 865ms, sequenceid=371, compaction requested=false 2023-04-05 18:54:51,282 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:51,282 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,282 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:51,282 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:51,282 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:51,285 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d87f452[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:51,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:51,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,290 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:51,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:51,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:51,290 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:51,291 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:51,292 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,292 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,292 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:51,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:51,300 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:51,300 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:51,300 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:51,300 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:51,300 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:51,302 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,302 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,302 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,303 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,308 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:51,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:51,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,308 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:51,308 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:51,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:51,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:51,346 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:54:51,347 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:51,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:51,347 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:51,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:51,347 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:51,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:51,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720951362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720951363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720951364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720951365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,408 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:51,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:51,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:51,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:51,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:51,412 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:51,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:51,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:51,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:51,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,414 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:51,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:51,416 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,416 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:51,416 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:51,416 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:51,418 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,418 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,418 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,418 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:51,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c86ecc8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:51,419 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:51,419 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,421 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:51,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:51,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,422 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:51,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:51,425 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:51,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:51,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:51,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:51,425 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:51,427 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,428 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,428 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,431 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,431 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,431 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:51,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:51,431 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,431 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:51,431 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,432 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:51,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,432 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720951466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720951467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720951467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720951468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,532 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:51,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:51,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:51,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:51,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:51,535 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:51,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:51,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:51,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:51,538 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,538 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,539 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:51,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:51,541 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,541 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:51,541 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:51,541 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:51,543 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:51,544 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:51,544 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bae6cf0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:51,546 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:51,546 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,546 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:51,547 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:51,548 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,548 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:51,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:51,550 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:51,550 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:51,550 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:51,550 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:51,550 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:51,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,552 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,553 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,555 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:51,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:51,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:51,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,555 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:51,556 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:51,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,656 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:51,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:51,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:51,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:51,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:51,659 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:51,659 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:51,659 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:51,659 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:51,660 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,660 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,661 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,661 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,662 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,662 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:51,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:51,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:51,663 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:51,663 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:51,663 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:51,663 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:51,663 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:51,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:51,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:51,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:51,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4209bcf3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:51,667 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:51,667 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,668 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,669 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:51,669 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:51,669 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:51,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:51,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720951669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720951670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720951670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720951671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,673 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:51,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:51,676 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:51,676 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:51,676 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:51,676 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:51,676 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:51,677 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,677 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,677 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,678 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,679 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,682 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,682 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,682 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,682 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:51,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:51,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,682 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,682 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,682 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,683 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,683 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,683 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:51,683 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:51,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/22450ea399164d01a55ee39720f5fbf4 2023-04-05 18:54:51,783 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:51,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:51,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:51,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:51,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:51,787 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:51,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:51,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:51,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:51,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,792 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,792 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,792 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:51,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:51,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:51,796 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,796 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:51,796 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:51,796 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:51,800 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,800 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:51,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@709ee584[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:51,803 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:51,803 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:51,804 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,804 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:51,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:51,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:51,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:51,810 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:51,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:51,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:51,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:51,810 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:51,812 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,812 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,812 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,816 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:51,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:51,822 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,822 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:51,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,822 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:51,922 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:51,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:51,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:51,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:51,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:51,931 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:51,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:51,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:51,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:51,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,932 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,932 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:51,934 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,934 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:51,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:51,935 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,935 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:51,935 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:51,935 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:51,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:51,937 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b95a7f3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:51,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:51,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,942 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:51,942 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,942 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:51,945 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:51,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:51,948 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:51,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:51,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:51,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:51,948 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:51,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:51,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,951 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,953 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,957 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:51,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:51,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:51,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:51,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:51,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:51,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,958 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:51,958 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:51,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:51,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:51,959 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:51,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:51,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720951972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720951973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720951974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:51,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:51,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720951974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,058 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:52,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:52,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:52,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:52,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:52,063 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:52,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:52,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:52,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,065 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,065 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,068 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,068 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:52,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:52,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:52,080 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,080 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:52,080 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:52,080 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:52,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:52,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1354919d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:52,087 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:52,087 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:52,089 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,089 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:52,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,091 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:52,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:52,100 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:52,100 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:52,100 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:52,100 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:52,100 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:52,102 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,104 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,105 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,111 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,111 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,111 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,111 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,111 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:52,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:52,111 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:52,112 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:52,112 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:52,111 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720952128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,132 DEBUG [Thread-1594] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:54:52,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/adc455bd936740729edc7d85ecebb33e 2023-04-05 18:54:52,212 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:52,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:52,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:52,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:52,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:52,224 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:52,224 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,224 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:52,224 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:52,224 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/7bb779bfa1d945649cc5ceefa0185bbc 2023-04-05 18:54:52,225 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,225 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,226 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,227 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:52,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:52,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:52,231 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,231 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:52,231 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:52,231 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:52,232 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,232 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:52,232 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:52,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/22450ea399164d01a55ee39720f5fbf4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/22450ea399164d01a55ee39720f5fbf4 2023-04-05 18:54:52,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,234 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,236 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,237 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:52,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/22450ea399164d01a55ee39720f5fbf4, entries=200, sequenceid=384, filesize=14.3 K 2023-04-05 18:54:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/adc455bd936740729edc7d85ecebb33e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/adc455bd936740729edc7d85ecebb33e 2023-04-05 18:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:52,239 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c354920[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:52,239 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,241 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:52,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:52,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:52,241 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,241 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:52,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/adc455bd936740729edc7d85ecebb33e, entries=150, sequenceid=384, filesize=11.9 K 2023-04-05 18:54:52,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/7bb779bfa1d945649cc5ceefa0185bbc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7bb779bfa1d945649cc5ceefa0185bbc 2023-04-05 18:54:52,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:52,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,247 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7bb779bfa1d945649cc5ceefa0185bbc, entries=150, sequenceid=384, filesize=11.9 K 2023-04-05 18:54:52,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,247 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c2dced816e56beff0d38901aab0de290 in 901ms, sequenceid=384, compaction requested=true 2023-04-05 18:54:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,248 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:52,248 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:52,248 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:52,248 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,248 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:52,248 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:52,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:52,248 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:52,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:52,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:52,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:52,249 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:52,249 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:52,249 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/B is initiating minor compaction (all files) 2023-04-05 18:54:52,249 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39306 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:52,249 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/B in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,249 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/A is initiating minor compaction (all files) 2023-04-05 18:54:52,249 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/0a1ba0c8feff4740a25cb117e199e134, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c49c2efefa0b417bb8f5978306daddc5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/adc455bd936740729edc7d85ecebb33e] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.0 K 2023-04-05 18:54:52,249 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/A in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,249 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/2ad042cbc18f4f00b9ee5873cf8ffee2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9807f17e20f4e38926d203787fcfbe6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/22450ea399164d01a55ee39720f5fbf4] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=38.4 K 2023-04-05 18:54:52,249 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0a1ba0c8feff4740a25cb117e199e134, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680720889023 2023-04-05 18:54:52,250 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 2ad042cbc18f4f00b9ee5873cf8ffee2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680720889023 2023-04-05 18:54:52,250 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting c49c2efefa0b417bb8f5978306daddc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680720889100 2023-04-05 18:54:52,250 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c9807f17e20f4e38926d203787fcfbe6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680720889100 2023-04-05 18:54:52,250 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting adc455bd936740729edc7d85ecebb33e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680720891233 2023-04-05 18:54:52,250 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 22450ea399164d01a55ee39720f5fbf4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680720891233 2023-04-05 18:54:52,251 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,251 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,253 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,256 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,256 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,256 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:52,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:52,256 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:52,256 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:52,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,256 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:52,257 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:52,256 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,261 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#B#compaction#233 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:52,269 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#A#compaction#234 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:52,315 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/a589866cc61640fe974cc2574a99e899 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/a589866cc61640fe974cc2574a99e899 2023-04-05 18:54:52,320 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/aba1447cad3a40b7be0713904f627cc7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/aba1447cad3a40b7be0713904f627cc7 2023-04-05 18:54:52,322 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/B of c2dced816e56beff0d38901aab0de290 into a589866cc61640fe974cc2574a99e899(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:52,322 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:52,322 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/B, priority=13, startTime=1680720892248; duration=0sec 2023-04-05 18:54:52,322 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:52,322 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:52,326 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:52,326 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/C is initiating minor compaction (all files) 2023-04-05 18:54:52,326 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/C in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,326 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7f7a3fd1922d40328177114a304b95ed, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/2fabf7552fb74a6a9828e5b2848742e2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7bb779bfa1d945649cc5ceefa0185bbc] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.0 K 2023-04-05 18:54:52,327 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 7f7a3fd1922d40328177114a304b95ed, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680720889023 2023-04-05 18:54:52,327 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 2fabf7552fb74a6a9828e5b2848742e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680720889100 2023-04-05 18:54:52,328 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 7bb779bfa1d945649cc5ceefa0185bbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680720891233 2023-04-05 18:54:52,336 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#C#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:52,339 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/A of c2dced816e56beff0d38901aab0de290 into aba1447cad3a40b7be0713904f627cc7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:52,339 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:52,339 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/A, priority=13, startTime=1680720892248; duration=0sec 2023-04-05 18:54:52,339 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:52,357 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:52,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:52,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:52,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:52,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:52,364 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:52,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:52,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:52,365 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,365 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,366 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,366 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:52,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:52,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:52,373 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/e9cc990c62554eb9a6fe47e889b4f419 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e9cc990c62554eb9a6fe47e889b4f419 2023-04-05 18:54:52,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:52,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:52,373 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,373 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:52,373 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:54:52,374 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:52,374 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:52,374 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:52,374 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:52,376 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:52,376 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:52,380 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/C of c2dced816e56beff0d38901aab0de290 into e9cc990c62554eb9a6fe47e889b4f419(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:52,380 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:52,380 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/C, priority=13, startTime=1680720892248; duration=0sec 2023-04-05 18:54:52,380 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:52,394 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/644c9dfad96940a1939d4a14bacc4bb8 2023-04-05 18:54:52,405 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/1631a6e438724c36bda0a8539a070c4f 2023-04-05 18:54:52,438 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/8485464c88b445849cc107916f9c74d4 2023-04-05 18:54:52,443 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/644c9dfad96940a1939d4a14bacc4bb8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/644c9dfad96940a1939d4a14bacc4bb8 2023-04-05 18:54:52,452 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/644c9dfad96940a1939d4a14bacc4bb8, entries=150, sequenceid=406, filesize=11.9 K 2023-04-05 18:54:52,454 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/1631a6e438724c36bda0a8539a070c4f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/1631a6e438724c36bda0a8539a070c4f 2023-04-05 18:54:52,462 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/1631a6e438724c36bda0a8539a070c4f, entries=150, sequenceid=406, filesize=11.9 K 2023-04-05 18:54:52,464 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/8485464c88b445849cc107916f9c74d4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/8485464c88b445849cc107916f9c74d4 2023-04-05 18:54:52,473 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/8485464c88b445849cc107916f9c74d4, entries=150, sequenceid=406, filesize=11.9 K 2023-04-05 18:54:52,473 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for c2dced816e56beff0d38901aab0de290 in 100ms, sequenceid=406, compaction requested=false 2023-04-05 18:54:52,474 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:52,474 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,474 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:52,474 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:52,474 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,476 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:52,476 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:52,487 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:54:52,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:52,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:52,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:52,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:52,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:52,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:52,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:52,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b003de7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:52,489 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:52,489 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,491 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,491 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:52,492 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:52,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:52,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:52,494 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,494 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:52,494 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,494 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:52,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:52,497 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:52,498 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:52,498 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:52,498 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:52,498 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:52,500 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,500 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,501 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,502 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,513 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:52,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:52,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,517 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:52,517 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:52,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680720892503, max timestamp=1680720892538Num uniques 859; 2023-04-05 18:54:52,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:54:52,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720892503, max timestamp=1680720892538Num uniques 859; 2023-04-05 18:54:52,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:54:52,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720892503, max timestamp=1680720892538Num uniques 859; 2023-04-05 18:54:52,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:54:52,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720952574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720952574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720952574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720952577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,617 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:52,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:52,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:52,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:52,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:52,621 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:52,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:52,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:52,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,622 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,623 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,623 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:52,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:52,628 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,628 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:52,628 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:52,628 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,630 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:52,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d9cbb4b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:52,634 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:52,634 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,635 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,635 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,635 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:52,641 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,642 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:52,642 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,642 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:52,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:52,644 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:52,644 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:52,644 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:52,644 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:52,644 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:52,646 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,646 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,646 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,648 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,654 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,654 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,654 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:52,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:52,654 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,654 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,654 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:52,654 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:52,654 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,654 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,655 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,655 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,655 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720952679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720952679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720952679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720952681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,754 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:52,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:52,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:52,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:52,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:52,758 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:52,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:52,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:52,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,760 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,760 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:52,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:52,762 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,762 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:52,762 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:52,763 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,764 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,765 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a730f1c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:52,766 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:52,766 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,767 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:52,767 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,767 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:52,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:52,776 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:52,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:52,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:52,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:52,776 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:52,783 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,783 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,783 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,783 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,784 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:52,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:52,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,787 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:52,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,787 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:52,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720952882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720952883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720952883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:52,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720952884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,887 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:52,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:52,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:52,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:52,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:52,891 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:52,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:52,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:52,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,892 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,892 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:52,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:52,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:52,894 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,894 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:52,894 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:52,894 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:52,896 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,897 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,897 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:52,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cb9293b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:52,898 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:52,898 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,900 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:52,902 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/e7b95641801549278c5979c39da69b71 2023-04-05 18:54:52,905 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,905 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,905 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,905 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:52,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:52,909 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:52,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:52,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:52,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:52,909 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:52,913 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,913 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,913 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:52,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:52,914 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/207696d261b34f74879f41c70277b55e 2023-04-05 18:54:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:52,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:52,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:52,921 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:52,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:52,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:52,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:52,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:52,922 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,922 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:52,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:52,922 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:52,922 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:52,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:52,922 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:52,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/03c4f2728b0f4bc7b39915471b0bc698 2023-04-05 18:54:52,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/e7b95641801549278c5979c39da69b71 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/e7b95641801549278c5979c39da69b71 2023-04-05 18:54:52,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/e7b95641801549278c5979c39da69b71, entries=300, sequenceid=417, filesize=19.0 K 2023-04-05 18:54:52,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/207696d261b34f74879f41c70277b55e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/207696d261b34f74879f41c70277b55e 2023-04-05 18:54:52,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/207696d261b34f74879f41c70277b55e, entries=150, sequenceid=417, filesize=11.9 K 2023-04-05 18:54:52,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/03c4f2728b0f4bc7b39915471b0bc698 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/03c4f2728b0f4bc7b39915471b0bc698 2023-04-05 18:54:52,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/03c4f2728b0f4bc7b39915471b0bc698, entries=150, sequenceid=417, filesize=11.9 K 2023-04-05 18:54:52,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c2dced816e56beff0d38901aab0de290 in 467ms, sequenceid=417, compaction requested=true 2023-04-05 18:54:52,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:52,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:52,954 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:52,954 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:52,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:52,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:52,955 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44288 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:52,955 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/A is initiating minor compaction (all files) 2023-04-05 18:54:52,955 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:52,955 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/A in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,955 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/B is initiating minor compaction (all files) 2023-04-05 18:54:52,955 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/aba1447cad3a40b7be0713904f627cc7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/644c9dfad96940a1939d4a14bacc4bb8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/e7b95641801549278c5979c39da69b71] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=43.3 K 2023-04-05 18:54:52,955 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/B in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:52,955 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/a589866cc61640fe974cc2574a99e899, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/1631a6e438724c36bda0a8539a070c4f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/207696d261b34f74879f41c70277b55e] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.1 K 2023-04-05 18:54:52,955 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting aba1447cad3a40b7be0713904f627cc7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680720891233 2023-04-05 18:54:52,956 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a589866cc61640fe974cc2574a99e899, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680720891233 2023-04-05 18:54:52,956 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 644c9dfad96940a1939d4a14bacc4bb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680720891360 2023-04-05 18:54:52,965 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1631a6e438724c36bda0a8539a070c4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680720891360 2023-04-05 18:54:52,965 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting e7b95641801549278c5979c39da69b71, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680720892478 2023-04-05 18:54:52,966 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 207696d261b34f74879f41c70277b55e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680720892481 2023-04-05 18:54:52,973 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#A#compaction#242 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:52,977 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#B#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:53,022 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:53,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:53,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:53,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:53,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:53,025 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:53,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:53,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:53,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:53,026 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:53,026 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:53,028 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,028 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:53,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:53,028 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,028 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:53,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:53,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:53,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:53,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:53,030 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:53,030 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:53,030 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-05 18:54:53,030 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:53,030 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:53,030 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:53,030 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:53,030 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:53,030 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:53,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:53,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:53,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720953192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720953193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34522 deadline: 1680720953195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720953196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34510 deadline: 1680720953297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720953298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720953301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720953301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,395 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/8b33f98b5d20491bbc8180d53c627b33 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8b33f98b5d20491bbc8180d53c627b33 2023-04-05 18:54:53,400 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/A of c2dced816e56beff0d38901aab0de290 into 8b33f98b5d20491bbc8180d53c627b33(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:53,400 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:53,400 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/A, priority=13, startTime=1680720892954; duration=0sec 2023-04-05 18:54:53,400 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:53,400 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:53,401 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:53,401 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/C is initiating minor compaction (all files) 2023-04-05 18:54:53,401 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/C in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:53,401 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e9cc990c62554eb9a6fe47e889b4f419, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/8485464c88b445849cc107916f9c74d4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/03c4f2728b0f4bc7b39915471b0bc698] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.1 K 2023-04-05 18:54:53,402 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting e9cc990c62554eb9a6fe47e889b4f419, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680720891233 2023-04-05 18:54:53,402 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 8485464c88b445849cc107916f9c74d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680720891360 2023-04-05 18:54:53,403 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 03c4f2728b0f4bc7b39915471b0bc698, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680720892481 2023-04-05 18:54:53,410 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#C#compaction#245 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:53,412 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/20c52f6e41b947b2a8aa862d3bf5e8ec as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/20c52f6e41b947b2a8aa862d3bf5e8ec 2023-04-05 18:54:53,418 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/B of c2dced816e56beff0d38901aab0de290 into 20c52f6e41b947b2a8aa862d3bf5e8ec(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:53,418 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:53,418 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/B, priority=13, startTime=1680720892954; duration=0sec 2023-04-05 18:54:53,418 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:53,423 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/bb7a1a4277f74d9592abc81c1c13539d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bb7a1a4277f74d9592abc81c1c13539d 2023-04-05 18:54:53,428 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/C of c2dced816e56beff0d38901aab0de290 into bb7a1a4277f74d9592abc81c1c13539d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:53,428 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:53,428 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/C, priority=13, startTime=1680720892954; duration=0sec 2023-04-05 18:54:53,428 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:53,438 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/8a4bbc25f7b24687ab766fbaa7a3ee4e 2023-04-05 18:54:53,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720953501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720953502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720953506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720953506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720953804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720953804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720953809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:53,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720953810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:53,850 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/895b27a065d247b3a8f5a93aa65e116a 2023-04-05 18:54:54,263 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/279ad0c2a68f4336bad299636a20d45c 2023-04-05 18:54:54,267 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/8a4bbc25f7b24687ab766fbaa7a3ee4e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8a4bbc25f7b24687ab766fbaa7a3ee4e 2023-04-05 18:54:54,271 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8a4bbc25f7b24687ab766fbaa7a3ee4e, entries=150, sequenceid=452, filesize=11.9 K 2023-04-05 18:54:54,271 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/895b27a065d247b3a8f5a93aa65e116a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/895b27a065d247b3a8f5a93aa65e116a 2023-04-05 18:54:54,275 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/895b27a065d247b3a8f5a93aa65e116a, entries=150, sequenceid=452, filesize=11.9 K 2023-04-05 18:54:54,276 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/279ad0c2a68f4336bad299636a20d45c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/279ad0c2a68f4336bad299636a20d45c 2023-04-05 18:54:54,279 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/279ad0c2a68f4336bad299636a20d45c, entries=150, sequenceid=452, filesize=11.9 K 2023-04-05 18:54:54,280 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for c2dced816e56beff0d38901aab0de290 in 1250ms, sequenceid=452, compaction requested=false 2023-04-05 18:54:54,280 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:54,280 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:54,282 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,282 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,282 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,283 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ccc64e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:54,284 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:54,284 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:54,285 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,285 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:54,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:54,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:54,289 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:54,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:54,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:54,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:54,290 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:54,292 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,292 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,292 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,293 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,294 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,297 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,297 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,297 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:54,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:54,297 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,297 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,297 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,297 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,297 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,297 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:54,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,297 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:54,297 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:54,308 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:54:54,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:54,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:54,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:54,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:54,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:54,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:54,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/edca4b38c9634ba4af269a572d6d3f65 2023-04-05 18:54:54,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680720894333, max timestamp=1680720894336Num uniques 859; 2023-04-05 18:54:54,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:54,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680720894333, max timestamp=1680720894336Num uniques 859; 2023-04-05 18:54:54,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:54,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680720894333, max timestamp=1680720894336Num uniques 859; 2023-04-05 18:54:54,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:54:54,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720954350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720954352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720954352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720954353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,397 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:54,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:54,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:54,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:54,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:54,401 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:54,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:54,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:54,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:54,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,405 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,405 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,405 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,405 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:54,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:54,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:54,406 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:54,406 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:54,406 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:54,406 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:54,407 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,407 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:54,407 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:54,407 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:54,409 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,410 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,410 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:54,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec2753f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:54,411 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:54,411 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,412 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:54,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,414 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:54,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:54,416 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:54,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:54,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:54,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:54,416 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:54,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,418 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,419 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,422 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:54,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:54,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:54,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,423 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:54,423 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:54,423 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720954454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720954458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720954458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720954458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,523 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:54,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:54,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:54,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:54,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:54,526 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:54,526 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:54,526 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:54,526 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:54,526 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,526 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,528 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,528 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,528 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,528 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:54,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:54,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:54,529 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,529 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:54,529 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:54,530 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:54,532 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,532 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ccb8e14[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:54,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:54,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,534 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:54,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,535 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:54,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:54,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,537 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:54,539 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:54,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:54,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:54,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:54,540 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:54,541 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,541 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,541 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,542 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,543 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,545 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:54,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:54,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,545 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:54,545 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:54,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,646 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:54,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:54,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:54,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:54,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:54,649 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:54,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:54,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:54,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:54,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,650 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:54,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:54,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:54,652 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,652 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:54,652 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:54,652 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:54,655 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,656 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,656 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19de9047[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:54,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:54,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,658 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,659 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:54,659 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:54,659 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:54,659 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,659 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,660 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720954657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:54,663 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:54,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:54,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:54,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:54,663 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:54,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720954661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720954661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720954661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,665 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,666 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:54,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:54,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:54,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,670 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:54,670 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:54,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,733 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/13ec9af0258244909fe4843a3d6099f7 2023-04-05 18:54:54,770 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:54,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:54,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:54,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:54,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:54,773 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:54,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:54,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:54,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:54,774 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,774 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,775 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,775 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,775 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,775 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:54,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:54,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:54,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:54,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:54,781 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,781 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,781 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,781 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49f0db84[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:54,783 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:54,783 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:54,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:54,785 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,785 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:54,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,786 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:54,789 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:54,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:54,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:54,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:54,789 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:54,791 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,791 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,791 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,791 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,793 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,795 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:54,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:54,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,796 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:54,796 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:54,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,896 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:54,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:54,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:54,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:54,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:54,900 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:54,900 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:54,900 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:54,900 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:54,900 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,900 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,903 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,903 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,904 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:54,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:54,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:54,906 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,906 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:54,906 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:54,906 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,908 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24762408[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:54,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:54,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,912 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:54,912 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,912 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:54,914 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:54,914 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,914 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,914 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:54,917 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:54,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:54,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:54,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:54,917 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:54,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:54,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:54,921 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:54,923 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:54,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:54,926 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:54,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:54,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:54,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:54,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:54,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:54,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:54,926 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:54,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:54,926 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:54,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:54,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720954961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720954965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720954967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:54,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:54,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720954967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,027 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:55,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:55,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:55,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:55,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:55,031 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:55,031 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:55,031 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:55,031 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:55,032 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,032 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:55,033 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:55,033 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:55,034 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,034 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:55,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:55,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:55,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:55,037 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:55,037 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:55,037 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:55,038 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:55,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:55,040 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:55,041 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:55,041 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:55,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:55,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46db5bc[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:55,044 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:55,044 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:55,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:55,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:55,046 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:55,046 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:55,046 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:55,046 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,046 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:55,048 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:55,049 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:55,049 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:55,053 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:55,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:55,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:55,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:55,054 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:55,056 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:55,056 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,056 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:55,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:55,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:55,060 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:55,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:55,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:55,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:55,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:55,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:55,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:55,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:55,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:55,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:55,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,064 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:55,064 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:55,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:55,142 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/bed199d3eac7458585391100a7b9e5be 2023-04-05 18:54:55,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/edca4b38c9634ba4af269a572d6d3f65 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/edca4b38c9634ba4af269a572d6d3f65 2023-04-05 18:54:55,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/edca4b38c9634ba4af269a572d6d3f65, entries=150, sequenceid=467, filesize=11.9 K 2023-04-05 18:54:55,151 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/13ec9af0258244909fe4843a3d6099f7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/13ec9af0258244909fe4843a3d6099f7 2023-04-05 18:54:55,155 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/13ec9af0258244909fe4843a3d6099f7, entries=150, sequenceid=467, filesize=11.9 K 2023-04-05 18:54:55,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/bed199d3eac7458585391100a7b9e5be as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bed199d3eac7458585391100a7b9e5be 2023-04-05 18:54:55,159 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bed199d3eac7458585391100a7b9e5be, entries=150, sequenceid=467, filesize=11.9 K 2023-04-05 18:54:55,160 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for c2dced816e56beff0d38901aab0de290 in 852ms, sequenceid=467, compaction requested=true 2023-04-05 18:54:55,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:55,160 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:55,160 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:55,160 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:55,160 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:55,160 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:55,161 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:55,161 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/A is initiating minor compaction (all files) 2023-04-05 18:54:55,161 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/A in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:55,161 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8b33f98b5d20491bbc8180d53c627b33, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8a4bbc25f7b24687ab766fbaa7a3ee4e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/edca4b38c9634ba4af269a572d6d3f65] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.2 K 2023-04-05 18:54:55,162 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:55,162 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/B is initiating minor compaction (all files) 2023-04-05 18:54:55,162 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 8b33f98b5d20491bbc8180d53c627b33, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680720892481 2023-04-05 18:54:55,162 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/B in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:55,162 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/20c52f6e41b947b2a8aa862d3bf5e8ec, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/895b27a065d247b3a8f5a93aa65e116a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/13ec9af0258244909fe4843a3d6099f7] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.2 K 2023-04-05 18:54:55,162 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 8a4bbc25f7b24687ab766fbaa7a3ee4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680720892544 2023-04-05 18:54:55,163 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 20c52f6e41b947b2a8aa862d3bf5e8ec, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680720892481 2023-04-05 18:54:55,163 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting edca4b38c9634ba4af269a572d6d3f65, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720893190 2023-04-05 18:54:55,163 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 895b27a065d247b3a8f5a93aa65e116a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680720892544 2023-04-05 18:54:55,163 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 13ec9af0258244909fe4843a3d6099f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720893190 2023-04-05 18:54:55,164 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:55,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:55,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:55,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:55,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:55,168 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:55,168 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:55,168 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:55,168 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:55,169 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,169 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:55,169 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#A#compaction#251 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:55,170 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:55,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:55,171 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#B#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:55,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:55,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:55,172 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:55,172 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,172 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:55,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:55,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:55,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:55,176 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:55,176 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:55,176 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:55,176 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:55,176 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-05 18:54:55,176 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:55,176 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:55,176 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:55,177 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:55,177 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:55,177 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:55,185 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/492e2a746e3a47c7a4b617c1a4f53028 2023-04-05 18:54:55,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:55,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:55,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720955475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720955477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720955478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720955479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,582 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/da1d12d0695743b6b48e90bebe1a3d25 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/da1d12d0695743b6b48e90bebe1a3d25 2023-04-05 18:54:55,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720955581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720955582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720955583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720955583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,586 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/3d7cf6a6e004425088a98ec7a48ad8ac as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/3d7cf6a6e004425088a98ec7a48ad8ac 2023-04-05 18:54:55,588 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/A of c2dced816e56beff0d38901aab0de290 into da1d12d0695743b6b48e90bebe1a3d25(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:55,588 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:55,588 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/A, priority=13, startTime=1680720895160; duration=0sec 2023-04-05 18:54:55,589 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:55,589 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:55,590 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:55,591 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/C is initiating minor compaction (all files) 2023-04-05 18:54:55,591 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/C in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:55,591 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bb7a1a4277f74d9592abc81c1c13539d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/279ad0c2a68f4336bad299636a20d45c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bed199d3eac7458585391100a7b9e5be] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.2 K 2023-04-05 18:54:55,591 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting bb7a1a4277f74d9592abc81c1c13539d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680720892481 2023-04-05 18:54:55,591 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 279ad0c2a68f4336bad299636a20d45c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680720892544 2023-04-05 18:54:55,592 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting bed199d3eac7458585391100a7b9e5be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720893190 2023-04-05 18:54:55,592 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/B of c2dced816e56beff0d38901aab0de290 into 3d7cf6a6e004425088a98ec7a48ad8ac(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:55,592 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:55,592 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/B, priority=13, startTime=1680720895160; duration=0sec 2023-04-05 18:54:55,592 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:55,598 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#C#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:55,600 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/6172690017134516924e7fd1633d2da5 2023-04-05 18:54:55,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720955785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720955785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720955786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:55,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:55,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720955786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,010 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/1ea4f0c4ecc24eedaf70e3cc21e56ee9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1ea4f0c4ecc24eedaf70e3cc21e56ee9 2023-04-05 18:54:56,016 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/7484e9580eb74b4890a0b17cc09257d3 2023-04-05 18:54:56,021 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/C of c2dced816e56beff0d38901aab0de290 into 1ea4f0c4ecc24eedaf70e3cc21e56ee9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:56,021 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:56,022 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/C, priority=13, startTime=1680720895160; duration=0sec 2023-04-05 18:54:56,022 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:56,023 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/492e2a746e3a47c7a4b617c1a4f53028 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/492e2a746e3a47c7a4b617c1a4f53028 2023-04-05 18:54:56,027 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/492e2a746e3a47c7a4b617c1a4f53028, entries=150, sequenceid=501, filesize=11.9 K 2023-04-05 18:54:56,028 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/6172690017134516924e7fd1633d2da5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/6172690017134516924e7fd1633d2da5 2023-04-05 18:54:56,056 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/6172690017134516924e7fd1633d2da5, entries=150, sequenceid=501, filesize=11.9 K 2023-04-05 18:54:56,057 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/7484e9580eb74b4890a0b17cc09257d3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7484e9580eb74b4890a0b17cc09257d3 2023-04-05 18:54:56,061 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7484e9580eb74b4890a0b17cc09257d3, entries=150, sequenceid=501, filesize=11.9 K 2023-04-05 18:54:56,062 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for c2dced816e56beff0d38901aab0de290 in 886ms, sequenceid=501, compaction requested=false 2023-04-05 18:54:56,062 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:56,062 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:56,065 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69dc51de[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:56,067 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:56,067 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:56,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:56,074 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:56,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:56,080 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:56,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:56,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:56,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:56,080 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:56,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:56,090 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 18:54:56,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:56,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:56,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:56,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:56,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:56,091 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:56,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,090 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:56,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:56,091 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:56,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:56,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720956106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720956107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720956109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720956112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34484 deadline: 1680720956172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,176 DEBUG [Thread-1594] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:54:56,191 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:56,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:56,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:56,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:56,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:56,194 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:56,194 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,194 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:56,194 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:56,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,197 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,197 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,198 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,198 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:56,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:56,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:56,199 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,199 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:56,199 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:56,199 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:56,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,202 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:56,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26bf8b9a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:56,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:56,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:56,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:56,208 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:56,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:56,212 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:56,213 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:56,213 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:56,213 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:56,213 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:56,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720956213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720956213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720956213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,215 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,215 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,215 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,217 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720956216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:56,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:56,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,220 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,220 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:56,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,220 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:56,321 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:56,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:56,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:56,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:56,324 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:56,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:56,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:56,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,327 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,328 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:56,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:56,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:56,331 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,331 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,331 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@337cbd21[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:56,342 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:56,342 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,343 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:56,343 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,344 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:56,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,345 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:56,345 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,345 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:56,348 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:56,348 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:56,348 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:56,348 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:56,348 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:56,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,351 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,353 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:56,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:56,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,356 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,357 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:56,357 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:56,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720956415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720956416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720956416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720956420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,457 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:56,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:56,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:56,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:56,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:56,460 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:56,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:56,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:56,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:56,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:56,465 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,465 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:56,465 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:56,465 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:56,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,467 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,467 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,467 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:56,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a553ea7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:56,469 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:56,469 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,471 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:56,471 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,471 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:56,473 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,473 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:56,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:56,475 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:56,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:56,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:56,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:56,475 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:56,481 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,481 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,481 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,482 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,483 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,486 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,486 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,486 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,487 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:56,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:56,487 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,487 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,487 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:56,487 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,487 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:56,487 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,487 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,487 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/8857acc64c3c4e448b622571759ecc43 2023-04-05 18:54:56,587 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:56,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:56,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:56,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:56,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:56,590 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:56,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:56,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:56,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,592 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,593 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,594 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,594 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:56,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:56,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:56,595 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,595 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:56,595 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:56,595 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:56,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,597 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,598 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,598 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@518cc692[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:56,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:56,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,601 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:56,601 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,601 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:56,608 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,608 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:56,608 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,608 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:56,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:56,610 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:56,610 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:56,610 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:56,610 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:56,610 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:56,612 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,612 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,613 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,614 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,617 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,617 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,617 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,617 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,612 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:56,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,617 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:56,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,620 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:56,621 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:56,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720956718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,721 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:56,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:56,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720956720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:56,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720956720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:56,724 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:56,724 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:56,724 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720956722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,724 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:56,725 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,725 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,726 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,726 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,727 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,727 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:56,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:56,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:56,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:56,728 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,728 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:56,728 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:56,728 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:56,731 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,731 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,731 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,731 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:56,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb56462[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:56,733 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:56,733 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,734 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,734 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:56,734 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:56,734 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:56,734 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,734 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:56,736 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:56,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:56,738 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:56,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:56,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:56,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:56,738 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:56,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,740 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,740 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,741 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:56,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:56,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:56,745 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,745 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:56,745 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:56,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,746 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,846 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:56,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:56,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:56,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:56,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:56,849 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:56,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:56,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:56,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,849 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,851 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,851 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,851 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,851 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:56,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:56,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:56,859 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,859 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:56,859 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:56,859 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:56,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,861 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:56,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d764580[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:56,863 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:56,863 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,864 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,864 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:56,864 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:56,864 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:56,864 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,864 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:56,867 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:56,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:56,869 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:56,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:56,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:56,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:56,869 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:56,871 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,871 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,871 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,872 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,873 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,879 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,879 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,879 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,879 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,879 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:56,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:56,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:56,879 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:56,880 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:56,880 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:56,880 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:56,880 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,880 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,880 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/8219d4430cc64bafb18158247e937f05 2023-04-05 18:54:56,980 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:56,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:56,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:56,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:56,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:56,984 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:56,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:56,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:56,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,984 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:56,986 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:56,986 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,986 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:56,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:56,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:56,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:56,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:56,991 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:56,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e25e88f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:56,991 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,993 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:56,994 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,994 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:56,996 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:56,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:56,998 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:56,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:56,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:56,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:56,998 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:57,000 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,000 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,000 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:57,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:57,001 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,002 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:57,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,005 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:57,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:57,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:57,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:57,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:57,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,006 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:57,006 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:57,006 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:57,106 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:57,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:57,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:57,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:57,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:57,110 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:57,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:57,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:57,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:57,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:57,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:57,112 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,113 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,113 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:57,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:57,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:57,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:57,114 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,114 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:57,114 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:57,114 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,116 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:57,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@192a138f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:57,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:57,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,119 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,119 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:57,119 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:57,119 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:57,120 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,120 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:57,121 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,121 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:57,121 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,121 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:57,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:57,123 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:57,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:57,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:57,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:57,124 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:57,126 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,126 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,126 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:57,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:57,127 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,128 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,130 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,130 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:57,130 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:57,130 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:57,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:57,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:57,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:57,130 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,131 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,131 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:57,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:57,131 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,131 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,131 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:57,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:57,131 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:57,131 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:57,233 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:57,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:57,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:57,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:57,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720957234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:57,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:57,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720957234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720957234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:57,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720957234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:57,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:57,237 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:57,237 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:57,237 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:57,237 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:57,238 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,238 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,239 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,239 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:57,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:57,239 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,239 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:57,240 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:57,240 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:57,241 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,241 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:57,241 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:57,241 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:57,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,242 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:57,244 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:57,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391cf2bc[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:57,244 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:57,247 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:57,247 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:57,247 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:57,248 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:57,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:57,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:57,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:57,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:57,251 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:57,251 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:57,251 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:57,251 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:57,251 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:57,252 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,252 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,252 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,253 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,254 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:57,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:57,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:57,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:57,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:57,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:57,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:57,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,257 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:57,257 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:57,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/1fa363bee5d6484e81e5bc297f4398e8 2023-04-05 18:54:57,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/8857acc64c3c4e448b622571759ecc43 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8857acc64c3c4e448b622571759ecc43 2023-04-05 18:54:57,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8857acc64c3c4e448b622571759ecc43, entries=150, sequenceid=518, filesize=11.9 K 2023-04-05 18:54:57,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/8219d4430cc64bafb18158247e937f05 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8219d4430cc64bafb18158247e937f05 2023-04-05 18:54:57,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8219d4430cc64bafb18158247e937f05, entries=150, sequenceid=518, filesize=11.9 K 2023-04-05 18:54:57,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/1fa363bee5d6484e81e5bc297f4398e8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1fa363bee5d6484e81e5bc297f4398e8 2023-04-05 18:54:57,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1fa363bee5d6484e81e5bc297f4398e8, entries=150, sequenceid=518, filesize=11.9 K 2023-04-05 18:54:57,338 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c2dced816e56beff0d38901aab0de290 in 1248ms, sequenceid=518, compaction requested=true 2023-04-05 18:54:57,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:57,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:57,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:57,338 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:57,338 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:57,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:57,339 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:57,339 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:57,340 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/A is initiating minor compaction (all files) 2023-04-05 18:54:57,340 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/B is initiating minor compaction (all files) 2023-04-05 18:54:57,340 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/A in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,340 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/B in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,340 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/da1d12d0695743b6b48e90bebe1a3d25, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/492e2a746e3a47c7a4b617c1a4f53028, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8857acc64c3c4e448b622571759ecc43] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.3 K 2023-04-05 18:54:57,340 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/3d7cf6a6e004425088a98ec7a48ad8ac, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/6172690017134516924e7fd1633d2da5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8219d4430cc64bafb18158247e937f05] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.3 K 2023-04-05 18:54:57,340 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting da1d12d0695743b6b48e90bebe1a3d25, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720893190 2023-04-05 18:54:57,340 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 3d7cf6a6e004425088a98ec7a48ad8ac, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720893190 2023-04-05 18:54:57,340 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 492e2a746e3a47c7a4b617c1a4f53028, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680720894342 2023-04-05 18:54:57,341 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 6172690017134516924e7fd1633d2da5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680720894342 2023-04-05 18:54:57,341 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 8857acc64c3c4e448b622571759ecc43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680720895469 2023-04-05 18:54:57,341 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 8219d4430cc64bafb18158247e937f05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680720895469 2023-04-05 18:54:57,347 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#B#compaction#260 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:57,349 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#A#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:57,356 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/2b3a2159664245f6b0100be1e75e39dc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b3a2159664245f6b0100be1e75e39dc 2023-04-05 18:54:57,357 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:57,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:57,360 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/d4ab5e8c7c084f988b26254cf74f0049 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/d4ab5e8c7c084f988b26254cf74f0049 2023-04-05 18:54:57,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:57,361 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:57,361 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:57,361 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:57,361 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:57,362 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,362 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,364 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/B of c2dced816e56beff0d38901aab0de290 into 2b3a2159664245f6b0100be1e75e39dc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:57,364 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:57,364 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/B, priority=13, startTime=1680720897338; duration=0sec 2023-04-05 18:54:57,364 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:57,364 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:57,366 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/A of c2dced816e56beff0d38901aab0de290 into d4ab5e8c7c084f988b26254cf74f0049(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:57,366 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:57,366 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/A, priority=13, startTime=1680720897338; duration=0sec 2023-04-05 18:54:57,366 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:57,366 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:57,367 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/C is initiating minor compaction (all files) 2023-04-05 18:54:57,367 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/C in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,367 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1ea4f0c4ecc24eedaf70e3cc21e56ee9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7484e9580eb74b4890a0b17cc09257d3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1fa363bee5d6484e81e5bc297f4398e8] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.3 K 2023-04-05 18:54:57,367 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1ea4f0c4ecc24eedaf70e3cc21e56ee9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720893190 2023-04-05 18:54:57,368 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 7484e9580eb74b4890a0b17cc09257d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680720894342 2023-04-05 18:54:57,368 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1fa363bee5d6484e81e5bc297f4398e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680720895469 2023-04-05 18:54:57,368 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:57,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:57,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:57,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:57,370 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:57,370 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:57,370 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:57,371 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:57,371 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:57,371 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,371 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:57,372 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 18:54:57,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:57,372 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:57,373 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:57,373 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:57,373 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:57,373 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:57,376 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#C#compaction#262 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:57,788 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/66ed109f1b92465ba0464ad435dc60a0 2023-04-05 18:54:57,792 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/47dbd76c2f3841c294238fa11a8f3e4b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/47dbd76c2f3841c294238fa11a8f3e4b 2023-04-05 18:54:57,798 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/C of c2dced816e56beff0d38901aab0de290 into 47dbd76c2f3841c294238fa11a8f3e4b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:57,798 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:57,798 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/C, priority=13, startTime=1680720897338; duration=0sec 2023-04-05 18:54:57,798 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/dcc76613ba2a435c987ea05689117de7 2023-04-05 18:54:57,798 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:57,809 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/eda1e8d629ae44b9b54b5ca783854f33 2023-04-05 18:54:57,813 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/66ed109f1b92465ba0464ad435dc60a0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/66ed109f1b92465ba0464ad435dc60a0 2023-04-05 18:54:57,817 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/66ed109f1b92465ba0464ad435dc60a0, entries=150, sequenceid=538, filesize=11.9 K 2023-04-05 18:54:57,817 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/dcc76613ba2a435c987ea05689117de7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/dcc76613ba2a435c987ea05689117de7 2023-04-05 18:54:57,826 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/dcc76613ba2a435c987ea05689117de7, entries=150, sequenceid=538, filesize=11.9 K 2023-04-05 18:54:57,827 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/eda1e8d629ae44b9b54b5ca783854f33 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/eda1e8d629ae44b9b54b5ca783854f33 2023-04-05 18:54:57,830 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/eda1e8d629ae44b9b54b5ca783854f33, entries=150, sequenceid=538, filesize=11.9 K 2023-04-05 18:54:57,831 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for c2dced816e56beff0d38901aab0de290 in 459ms, sequenceid=538, compaction requested=false 2023-04-05 18:54:57,831 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:57,831 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:57,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:57,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:57,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:57,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a9fa5a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:57,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:57,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:57,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:57,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:57,836 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,836 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:57,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:57,842 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:57,842 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:57,842 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:57,842 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:57,842 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:57,843 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,843 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,843 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:57,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,844 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,845 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,847 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:57,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:57,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:57,848 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:57,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:57,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:57,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:57,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,848 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:57,848 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:57,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,849 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:57,949 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:57,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:57,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:57,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:57,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:57,957 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:57,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:57,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:57,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:57,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,962 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:57,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:57,962 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,962 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,962 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:57,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:57,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:57,964 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,964 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:57,965 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:57,965 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:57,967 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57490190[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:57,970 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:57,970 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:57,971 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,971 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:57,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:57,972 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:57,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:57,977 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:57,977 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:57,978 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:57,978 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:57,978 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:57,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:57,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:57,980 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:57,980 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:57,982 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,990 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,990 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,990 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:57,990 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:57,990 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,990 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:57,990 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:57,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:57,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:57,990 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:57,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:57,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:57,990 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:57,991 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:57,991 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:57,991 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:57,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:57,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:57,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:58,091 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:58,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:58,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:58,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:58,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:58,095 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:58,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:58,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:58,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:58,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:58,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:58,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:58,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,098 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:58,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:58,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:58,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:58,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:58,099 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:58,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:58,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:58,099 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:58,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb6c69d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:58,103 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:58,103 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,104 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:58,105 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,105 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:58,108 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:58,108 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:58,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,108 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:58,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:58,110 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:58,111 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:58,111 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:58,111 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:58,111 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:58,113 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,113 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,113 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:58,113 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,114 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,117 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,117 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:58,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,120 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,120 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,120 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,120 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:58,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:58,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:58,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:58,120 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:58,120 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:58,120 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,121 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,121 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,121 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:58,221 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:58,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:58,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:58,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:58,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:58,224 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:58,224 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,224 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:58,224 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:58,225 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,225 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,226 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,226 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,227 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,227 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:58,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:58,229 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,229 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:58,229 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:58,229 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,240 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40614ee5[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:58,242 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:58,242 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:58,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:58,244 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:58,244 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,244 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:58,245 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,245 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:58,246 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:58,246 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,247 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:54:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:58,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:58,248 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:58,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:58,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:58,249 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:58,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:58,249 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:58,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:58,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:58,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:58,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:58,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:58,252 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,253 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,253 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,253 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:58,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/df1e401e19ed474680446ba9fd88d7ab 2023-04-05 18:54:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:58,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:58,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:58,259 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:58,259 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:58,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,260 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,260 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,260 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,260 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:58,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:58,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:58,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/92052549788c4f66807aceef8a751a17 2023-04-05 18:54:58,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720958268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720958268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720898264, max timestamp=1680720898271Num uniques 859; 2023-04-05 18:54:58,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:58,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720898264, max timestamp=1680720898271Num uniques 859; 2023-04-05 18:54:58,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:54:58,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720898264, max timestamp=1680720898271Num uniques 859; 2023-04-05 18:54:58,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 18:54:58,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720958298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720958299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,360 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:58,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:58,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:58,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:58,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:58,365 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:58,365 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,365 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:58,365 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:58,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,368 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:58,372 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,372 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:58,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:58,373 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,373 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:58,373 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:58,373 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:58,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:58,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:58,374 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,375 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720958373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:58,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720958374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59030799[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:58,378 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:58,378 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,382 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:58,382 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,382 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:58,384 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,384 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:58,384 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,384 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:58,388 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:58,388 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:58,388 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:58,388 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:58,388 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:58,391 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,391 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:58,391 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,394 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,397 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:58,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:58,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:58,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,398 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:58,398 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:58,398 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:58,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:58,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720958402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720958402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,498 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:58,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:58,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:58,501 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:58,502 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,502 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:58,502 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:58,502 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,502 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,504 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:58,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:58,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:58,505 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,505 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:58,505 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:58,505 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:58,507 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,507 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:58,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d9d8a8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:58,509 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:58,509 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,511 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,511 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:58,512 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,512 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:58,512 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:58,512 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:58,513 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,513 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:58,513 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,513 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:58,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:58,516 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:58,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:58,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:58,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:58,516 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:58,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,518 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,520 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,522 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,522 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,522 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,522 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,522 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,522 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:58,522 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,522 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:58,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:58,522 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:58,522 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:58,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,523 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:58,523 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:58,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:58,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720958578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720958578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720958606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720958606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,623 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:58,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:58,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:58,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:58,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:58,627 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:58,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:58,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:58,628 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,628 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,630 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,630 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,630 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,630 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:58,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:58,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:58,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:58,632 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,632 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:58,633 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:58,633 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:58,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:58,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e737cc0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:58,641 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:58,641 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,642 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,642 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:58,642 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:58,643 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:58,643 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,643 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:58,647 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:58,647 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,648 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,648 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:58,650 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:58,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:58,651 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:58,651 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:58,651 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:58,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,653 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:58,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,653 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:58,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:58,654 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:58,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:58,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:58,670 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:58,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:58,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,670 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:58,670 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:58,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,670 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:58,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:58,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:58,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:58,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/3c67746109d0483a82252c583f857126 2023-04-05 18:54:58,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/df1e401e19ed474680446ba9fd88d7ab as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df1e401e19ed474680446ba9fd88d7ab 2023-04-05 18:54:58,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df1e401e19ed474680446ba9fd88d7ab, entries=200, sequenceid=553, filesize=14.3 K 2023-04-05 18:54:58,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/92052549788c4f66807aceef8a751a17 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/92052549788c4f66807aceef8a751a17 2023-04-05 18:54:58,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/92052549788c4f66807aceef8a751a17, entries=150, sequenceid=553, filesize=11.9 K 2023-04-05 18:54:58,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/3c67746109d0483a82252c583f857126 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/3c67746109d0483a82252c583f857126 2023-04-05 18:54:58,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/3c67746109d0483a82252c583f857126, entries=150, sequenceid=553, filesize=11.9 K 2023-04-05 18:54:58,701 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.08 KB/130125 for c2dced816e56beff0d38901aab0de290 in 454ms, sequenceid=553, compaction requested=true 2023-04-05 18:54:58,701 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:58,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:58,702 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:58,702 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:58,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:58,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:58,703 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39714 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:58,703 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/A is initiating minor compaction (all files) 2023-04-05 18:54:58,703 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/A in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,703 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/d4ab5e8c7c084f988b26254cf74f0049, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/66ed109f1b92465ba0464ad435dc60a0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df1e401e19ed474680446ba9fd88d7ab] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=38.8 K 2023-04-05 18:54:58,703 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:58,703 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting d4ab5e8c7c084f988b26254cf74f0049, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680720895469 2023-04-05 18:54:58,704 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/B is initiating minor compaction (all files) 2023-04-05 18:54:58,704 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/B in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,704 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b3a2159664245f6b0100be1e75e39dc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/dcc76613ba2a435c987ea05689117de7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/92052549788c4f66807aceef8a751a17] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.4 K 2023-04-05 18:54:58,704 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 66ed109f1b92465ba0464ad435dc60a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1680720896106 2023-04-05 18:54:58,707 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting df1e401e19ed474680446ba9fd88d7ab, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680720898244 2023-04-05 18:54:58,708 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 2b3a2159664245f6b0100be1e75e39dc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680720895469 2023-04-05 18:54:58,708 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting dcc76613ba2a435c987ea05689117de7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1680720896106 2023-04-05 18:54:58,709 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 92052549788c4f66807aceef8a751a17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680720898245 2023-04-05 18:54:58,729 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#A#compaction#269 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:58,733 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#B#compaction#270 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:58,770 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:58,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:58,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:58,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:58,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:58,789 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:58,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:58,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:58,790 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,790 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,792 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:58,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:58,793 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,793 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:58,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:58,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:58,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:58,820 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:58,820 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:58,820 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:58,827 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/9cf625d712e04a4ca51a59eb5e685cdf as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9cf625d712e04a4ca51a59eb5e685cdf 2023-04-05 18:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:58,835 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,835 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:58,835 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/bb79d19d4c0643ca8ce0498a78713583 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bb79d19d4c0643ca8ce0498a78713583 2023-04-05 18:54:58,835 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=127.08 KB heapSize=364.15 KB 2023-04-05 18:54:58,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:58,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:58,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:58,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:58,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:58,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:54:58,840 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/A of c2dced816e56beff0d38901aab0de290 into 9cf625d712e04a4ca51a59eb5e685cdf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:58,840 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:58,840 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/A, priority=13, startTime=1680720898701; duration=0sec 2023-04-05 18:54:58,840 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:54:58,840 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:54:58,843 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:54:58,843 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/C is initiating minor compaction (all files) 2023-04-05 18:54:58,843 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/C in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:58,844 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/47dbd76c2f3841c294238fa11a8f3e4b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/eda1e8d629ae44b9b54b5ca783854f33, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/3c67746109d0483a82252c583f857126] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=36.4 K 2023-04-05 18:54:58,844 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 47dbd76c2f3841c294238fa11a8f3e4b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680720895469 2023-04-05 18:54:58,845 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting eda1e8d629ae44b9b54b5ca783854f33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1680720896106 2023-04-05 18:54:58,845 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 3c67746109d0483a82252c583f857126, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680720898245 2023-04-05 18:54:58,858 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/B of c2dced816e56beff0d38901aab0de290 into bb79d19d4c0643ca8ce0498a78713583(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:58,858 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:58,858 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/B, priority=13, startTime=1680720898702; duration=0sec 2023-04-05 18:54:58,858 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:58,864 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/c9bfd8b4333f46978e5f39e86d42fb95 2023-04-05 18:54:58,865 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#C#compaction#272 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:54:58,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:54:58,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:54:58,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720958934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720958934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720958934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:58,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:58,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720958935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720959039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720959039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720959040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720959040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720959242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720959242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720959243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720959243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,302 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/5c50569f54874db7ac15bb4f5fdf8098 2023-04-05 18:54:59,317 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/46fea136538c453ea22a9806093fdfac as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/46fea136538c453ea22a9806093fdfac 2023-04-05 18:54:59,321 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c2dced816e56beff0d38901aab0de290/C of c2dced816e56beff0d38901aab0de290 into 46fea136538c453ea22a9806093fdfac(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:54:59,321 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:59,321 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/C, priority=13, startTime=1680720898702; duration=0sec 2023-04-05 18:54:59,321 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:54:59,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720959545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720959546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720959546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:54:59,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720959546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,716 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/d8ae542492214dd79293eb519d25d918 2023-04-05 18:54:59,721 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/c9bfd8b4333f46978e5f39e86d42fb95 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9bfd8b4333f46978e5f39e86d42fb95 2023-04-05 18:54:59,726 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9bfd8b4333f46978e5f39e86d42fb95, entries=150, sequenceid=589, filesize=11.9 K 2023-04-05 18:54:59,728 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/5c50569f54874db7ac15bb4f5fdf8098 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5c50569f54874db7ac15bb4f5fdf8098 2023-04-05 18:54:59,733 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5c50569f54874db7ac15bb4f5fdf8098, entries=150, sequenceid=589, filesize=11.9 K 2023-04-05 18:54:59,734 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/d8ae542492214dd79293eb519d25d918 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/d8ae542492214dd79293eb519d25d918 2023-04-05 18:54:59,745 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/d8ae542492214dd79293eb519d25d918, entries=150, sequenceid=589, filesize=11.9 K 2023-04-05 18:54:59,746 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.08 KB/130125, heapSize ~364.10 KB/372840, currentSize=67.09 KB/68700 for c2dced816e56beff0d38901aab0de290 in 911ms, sequenceid=589, compaction requested=false 2023-04-05 18:54:59,746 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:54:59,746 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:54:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:54:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:54:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:54:59,748 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:54:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24af018d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:54:59,751 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:54:59,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:59,753 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:59,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:54:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:54:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:54:59,754 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,754 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:54:59,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:54:59,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:59,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:54:59,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:54:59,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:54:59,758 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:54:59,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:54:59,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:54:59,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:54:59,758 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:54:59,761 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:59,761 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:59,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:59,761 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:54:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:54:59,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:54:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:59,764 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:54:59,765 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:54:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:54:59,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:59,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:59,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:54:59,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:59,768 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:54:59,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:59,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:54:59,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:54:59,768 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:59,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:59,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:54:59,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:59,769 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:54:59,769 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:54:59,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:54:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:54:59,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:54:59,869 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:54:59,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:54:59,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:54:59,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:54:59,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:54:59,873 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:54:59,873 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:59,873 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:54:59,873 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:54:59,873 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:59,873 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:59,875 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:54:59,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:54:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:54:59,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:54:59,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:54:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:59,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:54:59,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:54:59,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:54:59,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:54:59,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:54:59,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:54:59,876 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:54:59,876 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:54:59,876 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:54:59,876 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:54:59,877 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:54:59,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:54:59,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:54:59,877 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 18:54:59,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:54:59,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:59,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:54:59,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:59,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:54:59,877 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:54:59,884 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/df14e3d744124ef8b19e96940949ceca 2023-04-05 18:54:59,896 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/bae561a1931748d2947fcb2ac426548e 2023-04-05 18:55:00,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:55:00,051 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:00,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720960063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720960064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720960064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720960066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720960168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720960168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720960168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720960169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,305 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/9080d679280d4421ab5c50ee920c6c80 2023-04-05 18:55:00,323 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/df14e3d744124ef8b19e96940949ceca as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df14e3d744124ef8b19e96940949ceca 2023-04-05 18:55:00,328 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df14e3d744124ef8b19e96940949ceca, entries=150, sequenceid=605, filesize=11.9 K 2023-04-05 18:55:00,329 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/bae561a1931748d2947fcb2ac426548e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bae561a1931748d2947fcb2ac426548e 2023-04-05 18:55:00,351 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bae561a1931748d2947fcb2ac426548e, entries=150, sequenceid=605, filesize=11.9 K 2023-04-05 18:55:00,353 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/9080d679280d4421ab5c50ee920c6c80 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/9080d679280d4421ab5c50ee920c6c80 2023-04-05 18:55:00,363 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/9080d679280d4421ab5c50ee920c6c80, entries=150, sequenceid=605, filesize=11.9 K 2023-04-05 18:55:00,364 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c2dced816e56beff0d38901aab0de290 in 487ms, sequenceid=605, compaction requested=true 2023-04-05 18:55:00,364 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:00,364 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,364 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:00,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:00,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:00,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:00,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6551e54e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:00,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,373 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:00,373 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:00,375 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:55:00,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:55:00,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:00,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:55:00,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:00,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:55:00,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:00,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:55:00,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:00,380 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,380 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:00,382 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:00,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:00,382 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:00,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:00,385 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:00,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:00,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:00,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:00,386 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:00,388 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,388 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:00,388 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,390 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720960387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720960387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,396 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:00,396 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:00,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:00,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:00,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:00,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:00,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:00,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,397 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:00,397 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:00,397 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:00,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:00,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720960392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720960393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,449 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/315ecf69713445ea90788c71e3c74e34 2023-04-05 18:55:00,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720960494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,495 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/331fe60caa0342509e06cfd109aa39f9 2023-04-05 18:55:00,497 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:00,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:00,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720960497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:00,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720960499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720960499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:00,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:00,504 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:00,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:00,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:00,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:00,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,507 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:00,507 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,507 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:00,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:00,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:00,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:00,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:00,528 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,528 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:00,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:00,528 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:00,528 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:00,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:00,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:00,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:00,530 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,530 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc56dd4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:00,532 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:00,532 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,533 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:00,534 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,534 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:00,535 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:00,536 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:00,538 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:00,538 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:00,538 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:00,538 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:00,538 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:00,546 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,546 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,546 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:00,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:00,550 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:00,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:00,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:00,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:00,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:00,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:00,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,551 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:00,551 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:00,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:00,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,651 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:00,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:00,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:00,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:00,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:00,654 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:00,654 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:00,654 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:00,654 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:00,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,655 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:00,656 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:00,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:00,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:00,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:00,658 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,658 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:00,658 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:00,658 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:00,660 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,661 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,661 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:00,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18a599d0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:00,662 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:00,662 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:00,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:00,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:00,665 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,665 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:00,666 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,666 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,666 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:00,666 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:00,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:00,669 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:00,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:00,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:00,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:00,669 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:00,671 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,671 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,671 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:00,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:00,672 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,675 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,675 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:00,675 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:00,675 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,676 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,676 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,676 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:00,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:00,675 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:00,675 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:00,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:00,676 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:00,676 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:00,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:00,676 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720960698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720960701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720960702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:00,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720960704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,776 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:00,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:00,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:00,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:00,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:00,779 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:00,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:00,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:00,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:00,780 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,780 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,782 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:00,782 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:00,782 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,782 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:00,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:00,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:00,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:00,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:00,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:00,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:00,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:00,783 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,783 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:00,783 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:00,783 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:00,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,786 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,786 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44da5020[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:00,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:00,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:00,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:00,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:00,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:00,790 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:00,793 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:00,793 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:00,793 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:00,793 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:00,793 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:00,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:00,796 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,797 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:00,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,799 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:00,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:00,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:00,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,800 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,800 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,800 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,800 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:00,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:00,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:00,800 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:00,800 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:00,900 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:00,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:00,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:00,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:00,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:00,904 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:00,904 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:00,904 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:00,904 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:00,904 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,904 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,905 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:00,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:00,906 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,906 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:00,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:00,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:00,908 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,908 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:00,908 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:00,908 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:00,910 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e4ba660[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:00,912 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:00,912 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,914 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:00,914 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,914 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:00,915 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,915 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,915 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:00,915 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:00,918 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:00,918 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:00,918 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:00,918 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:00,918 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:00,919 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,919 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:00,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,919 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:00,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:00,920 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,921 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/e7c3b05b3df3470597e6c785d70ac539 2023-04-05 18:55:00,929 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,929 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,929 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:00,929 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:00,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:00,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:00,929 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:00,929 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,929 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:00,929 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:00,929 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:00,929 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:00,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/315ecf69713445ea90788c71e3c74e34 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/315ecf69713445ea90788c71e3c74e34 2023-04-05 18:55:00,936 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:00,936 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:00,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:00,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:00,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/315ecf69713445ea90788c71e3c74e34, entries=200, sequenceid=625, filesize=14.3 K 2023-04-05 18:55:00,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/331fe60caa0342509e06cfd109aa39f9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/331fe60caa0342509e06cfd109aa39f9 2023-04-05 18:55:00,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/331fe60caa0342509e06cfd109aa39f9, entries=150, sequenceid=625, filesize=11.9 K 2023-04-05 18:55:00,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/e7c3b05b3df3470597e6c785d70ac539 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e7c3b05b3df3470597e6c785d70ac539 2023-04-05 18:55:00,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e7c3b05b3df3470597e6c785d70ac539, entries=150, sequenceid=625, filesize=11.9 K 2023-04-05 18:55:00,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for c2dced816e56beff0d38901aab0de290 in 579ms, sequenceid=625, compaction requested=true 2023-04-05 18:55:00,955 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:00,955 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:00,955 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:00,955 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:00,955 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:00,955 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:00,957 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52002 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:00,957 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/A is initiating minor compaction (all files) 2023-04-05 18:55:00,957 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/A in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,957 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9cf625d712e04a4ca51a59eb5e685cdf, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9bfd8b4333f46978e5f39e86d42fb95, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df14e3d744124ef8b19e96940949ceca, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/315ecf69713445ea90788c71e3c74e34] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=50.8 K 2023-04-05 18:55:00,957 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:00,957 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/B is initiating minor compaction (all files) 2023-04-05 18:55:00,957 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/B in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:00,957 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bb79d19d4c0643ca8ce0498a78713583, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5c50569f54874db7ac15bb4f5fdf8098, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bae561a1931748d2947fcb2ac426548e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/331fe60caa0342509e06cfd109aa39f9] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=48.4 K 2023-04-05 18:55:00,958 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 9cf625d712e04a4ca51a59eb5e685cdf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680720898245 2023-04-05 18:55:00,958 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c9bfd8b4333f46978e5f39e86d42fb95, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1680720898285 2023-04-05 18:55:00,958 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting bb79d19d4c0643ca8ce0498a78713583, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680720898245 2023-04-05 18:55:00,958 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting df14e3d744124ef8b19e96940949ceca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1680720898893 2023-04-05 18:55:00,959 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 5c50569f54874db7ac15bb4f5fdf8098, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1680720898285 2023-04-05 18:55:00,959 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 315ecf69713445ea90788c71e3c74e34, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680720900061 2023-04-05 18:55:00,959 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting bae561a1931748d2947fcb2ac426548e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1680720898893 2023-04-05 18:55:00,960 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 331fe60caa0342509e06cfd109aa39f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680720900061 2023-04-05 18:55:00,968 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#A#compaction#281 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:00,985 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#B#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:01,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:55:01,006 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 18:55:01,006 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:55:01,006 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:01,006 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:55:01,006 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:01,006 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:55:01,006 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:01,009 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/497014520cdf4619a201203735a5b771 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/497014520cdf4619a201203735a5b771 2023-04-05 18:55:01,017 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2dced816e56beff0d38901aab0de290/A of c2dced816e56beff0d38901aab0de290 into 497014520cdf4619a201203735a5b771(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:01,017 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:01,017 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/A, priority=12, startTime=1680720900955; duration=0sec 2023-04-05 18:55:01,017 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:01,017 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:01,019 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:01,019 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/C is initiating minor compaction (all files) 2023-04-05 18:55:01,019 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/C in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,019 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/46fea136538c453ea22a9806093fdfac, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/d8ae542492214dd79293eb519d25d918, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/9080d679280d4421ab5c50ee920c6c80, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e7c3b05b3df3470597e6c785d70ac539] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=48.4 K 2023-04-05 18:55:01,020 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 46fea136538c453ea22a9806093fdfac, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680720898245 2023-04-05 18:55:01,020 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting d8ae542492214dd79293eb519d25d918, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1680720898285 2023-04-05 18:55:01,021 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 9080d679280d4421ab5c50ee920c6c80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1680720898893 2023-04-05 18:55:01,021 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting e7c3b05b3df3470597e6c785d70ac539, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680720900061 2023-04-05 18:55:01,029 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:01,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:01,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:01,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720961037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720961037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720961038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720961039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,042 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#C#compaction#284 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:01,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:01,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:01,056 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:01,056 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,056 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:01,056 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:01,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,058 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:01,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:01,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:01,064 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,065 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:01,065 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:01,065 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:01,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:01,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:01,065 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:01,067 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,067 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,068 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,068 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:01,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6826c5b0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:01,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:01,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,071 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,071 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,071 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:01,071 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:01,071 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:01,071 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:01,071 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/80baae30296d4731a5ba7f10bd9137cf as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/80baae30296d4731a5ba7f10bd9137cf 2023-04-05 18:55:01,074 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:01,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,075 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2dced816e56beff0d38901aab0de290/C of c2dced816e56beff0d38901aab0de290 into 80baae30296d4731a5ba7f10bd9137cf(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:01,076 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:01,076 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/C, priority=12, startTime=1680720900955; duration=0sec 2023-04-05 18:55:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,076 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:01,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:01,077 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:01,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:01,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:01,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:01,077 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:01,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,081 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,082 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,085 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:01,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:01,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,085 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:01,085 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:01,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:01,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720961142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720961142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720961142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720961143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,185 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:01,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:01,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:01,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:01,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:01,189 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:01,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:01,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:01,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,190 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,190 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:01,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:01,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:01,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:01,192 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,192 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:01,192 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:01,192 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:01,192 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:01,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:01,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c79aa7d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:01,198 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:01,198 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:01,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:01,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:01,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:01,200 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,200 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:01,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:01,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:01,203 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:01,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:01,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:01,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:01,203 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:01,205 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,205 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,205 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:01,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:01,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:01,210 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,210 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:01,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,210 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:01,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,310 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:01,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:01,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:01,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:01,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:01,314 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:01,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:01,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:01,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,316 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:01,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:01,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:01,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:01,317 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,317 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:01,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:01,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:01,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:01,319 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:01,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a34ba82[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:01,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:01,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,322 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,322 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:01,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:01,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:01,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:01,327 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:01,327 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,327 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,327 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:01,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:01,329 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:01,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:01,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:01,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:01,329 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:01,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,332 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,333 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,336 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,337 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:01,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:01,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,337 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:01,337 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:01,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720961345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720961345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720961346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720961346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,411 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/813c48cc885d42dd91f400a204085ec6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/813c48cc885d42dd91f400a204085ec6 2023-04-05 18:55:01,417 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2dced816e56beff0d38901aab0de290/B of c2dced816e56beff0d38901aab0de290 into 813c48cc885d42dd91f400a204085ec6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:01,417 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:01,417 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/B, priority=12, startTime=1680720900955; duration=0sec 2023-04-05 18:55:01,417 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:01,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/7cf388e79bb4420588ac9b67a363d3d1 2023-04-05 18:55:01,437 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:01,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:01,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:01,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:01,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:01,441 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:01,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:01,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:01,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,453 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,453 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:01,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:01,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:01,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:01,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:01,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:01,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:01,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:01,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:01,454 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,454 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:01,454 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:01,454 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:01,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:01,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:01,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:01,456 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,456 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,457 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,457 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:01,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393f7d02[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:01,458 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:01,458 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,459 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:01,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:01,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:01,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:01,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:01,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:01,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:01,464 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:01,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:01,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:01,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:01,464 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:01,465 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,465 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,465 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,466 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,469 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:01,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:01,470 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,470 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:01,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,470 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:01,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:01,570 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:01,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:01,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:01,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:01,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:01,573 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:01,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:01,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:01,574 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,574 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:01,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:01,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:01,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:01,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:01,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:01,577 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,577 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:01,577 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:01,577 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:01,580 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:01,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:01,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ea24d0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:01,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:01,582 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,583 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:01,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:01,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:01,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:01,584 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:01,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:01,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:01,586 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:01,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:01,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:01,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:01,586 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:01,587 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,587 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,587 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,589 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,592 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:01,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:01,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:01,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,593 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:01,593 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:01,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720961649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720961650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:01,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720961650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720961650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,693 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:01,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:01,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:01,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:01,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:01,696 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:01,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:01,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:01,697 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,697 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,698 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:01,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:01,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:01,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:01,699 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:01,699 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,699 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:01,699 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:01,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:01,700 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,700 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:01,700 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:01,700 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:01,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:01,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:01,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:01,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:01,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76d14012[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:01,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:01,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:01,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:01,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:01,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:01,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:01,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:01,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:01,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:01,709 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:01,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:01,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:01,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:01,709 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:01,711 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,711 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,711 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,714 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:01,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:01,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,715 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,715 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:01,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:01,715 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:01,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,815 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:01,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:01,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:01,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:01,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:01,819 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:01,819 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,819 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:01,819 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:01,819 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,819 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,821 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:01,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:01,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:01,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:01,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:01,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:01,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:01,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:01,823 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,823 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:01,823 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:01,823 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:01,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:01,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:01,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:01,828 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,829 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,829 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:01,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195e04e0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:01,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:01,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,831 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:01,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:01,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:01,831 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,831 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:01,832 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:01,832 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,833 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:01,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:01,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:01,837 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:01,837 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:01,837 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:01,837 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:01,837 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:01,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,841 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,842 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/963e083961494254b5638e3484d18777 2023-04-05 18:55:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,843 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,846 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:01,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:01,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,847 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:01,847 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:01,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:01,847 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,847 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,947 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:01,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:01,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:01,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:01,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:01,960 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:01,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:01,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:01,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,961 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,962 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,962 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:01,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:01,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:01,963 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,963 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:01,964 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:01,964 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:01,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:01,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:01,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:01,964 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:01,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,965 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:01,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bcbbda4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:01,967 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:01,967 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,969 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,969 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:01,969 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,970 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:01,969 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:01,970 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:01,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:01,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:01,973 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:01,973 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:01,973 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:01,974 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:01,974 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:01,975 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,975 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:01,975 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:01,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,976 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,978 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,978 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,978 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:01,978 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:01,978 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:01,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:01,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:01,978 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:01,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:01,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:01,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:01,979 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:01,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:01,979 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:01,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:01,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:02,079 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:02,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:02,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:02,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:02,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:02,084 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:02,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:02,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:02,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:02,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,086 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:02,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:02,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:02,087 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,087 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:02,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:02,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:02,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:02,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:02,088 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:02,088 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:02,088 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:02,088 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:02,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,090 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,090 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,090 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:02,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d31dea9[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:02,091 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:02,092 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,092 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:02,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:02,094 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,094 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:02,094 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:02,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,094 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:02,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:02,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:02,096 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:02,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:02,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:02,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:02,097 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:02,099 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,099 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,099 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:02,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,100 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,101 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:02,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,103 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:02,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:02,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:02,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:02,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:02,103 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:02,103 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:02,103 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:02,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:02,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:02,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720962153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:02,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720962153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:02,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720962154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:02,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720962156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,203 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:02,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:02,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:02,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:02,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:02,206 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:02,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:02,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:02,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:02,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,208 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:02,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,209 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,209 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:02,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:02,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:02,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:02,210 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:02,210 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:02,210 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:02,210 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:02,212 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,212 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,212 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,212 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:02,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@111cd2cd[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:02,214 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:02,214 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,216 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:02,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:02,219 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,220 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:02,220 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:02,220 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:02,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:02,222 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:02,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:02,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:02,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:02,222 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:02,224 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,224 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,224 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:02,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,224 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,226 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,229 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,229 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,229 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,229 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,229 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,229 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,229 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,229 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:02,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:02,230 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:02,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:02,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:02,230 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:02,230 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:02,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:02,230 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:02,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:02,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/95f40f7880714fd8bd6afdf2d302f8f6 2023-04-05 18:55:02,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/7cf388e79bb4420588ac9b67a363d3d1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/7cf388e79bb4420588ac9b67a363d3d1 2023-04-05 18:55:02,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/7cf388e79bb4420588ac9b67a363d3d1, entries=200, sequenceid=639, filesize=14.3 K 2023-04-05 18:55:02,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/963e083961494254b5638e3484d18777 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/963e083961494254b5638e3484d18777 2023-04-05 18:55:02,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/963e083961494254b5638e3484d18777, entries=150, sequenceid=639, filesize=11.9 K 2023-04-05 18:55:02,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/95f40f7880714fd8bd6afdf2d302f8f6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/95f40f7880714fd8bd6afdf2d302f8f6 2023-04-05 18:55:02,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/95f40f7880714fd8bd6afdf2d302f8f6, entries=150, sequenceid=639, filesize=11.9 K 2023-04-05 18:55:02,271 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c2dced816e56beff0d38901aab0de290 in 1266ms, sequenceid=639, compaction requested=false 2023-04-05 18:55:02,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:02,330 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:02,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:02,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:02,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:02,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:02,333 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:02,334 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:02,334 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:02,334 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:02,334 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,334 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,335 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:02,335 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:02,336 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,336 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:02,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:02,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:02,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:02,337 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:02,337 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:02,337 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:02,337 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:02,337 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:02,337 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:02,337 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:02,337 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 18:55:02,338 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:55:02,338 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:02,338 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:55:02,338 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:02,338 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:55:02,338 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:02,746 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/b60b4cf6c364439d8572eb1d0d7fa188 2023-04-05 18:55:02,756 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/5e24b53d3d574693aacc22f0136fbaa4 2023-04-05 18:55:02,765 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/56d761f16605438994996e365ea93aac 2023-04-05 18:55:02,769 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/b60b4cf6c364439d8572eb1d0d7fa188 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/b60b4cf6c364439d8572eb1d0d7fa188 2023-04-05 18:55:02,773 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/b60b4cf6c364439d8572eb1d0d7fa188, entries=150, sequenceid=660, filesize=11.9 K 2023-04-05 18:55:02,773 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/5e24b53d3d574693aacc22f0136fbaa4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5e24b53d3d574693aacc22f0136fbaa4 2023-04-05 18:55:02,777 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5e24b53d3d574693aacc22f0136fbaa4, entries=150, sequenceid=660, filesize=11.9 K 2023-04-05 18:55:02,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/56d761f16605438994996e365ea93aac as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/56d761f16605438994996e365ea93aac 2023-04-05 18:55:02,780 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/56d761f16605438994996e365ea93aac, entries=150, sequenceid=660, filesize=11.9 K 2023-04-05 18:55:02,792 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for c2dced816e56beff0d38901aab0de290 in 455ms, sequenceid=660, compaction requested=true 2023-04-05 18:55:02,792 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:02,792 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:02,794 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:02,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@788cbd0a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:02,797 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:02,797 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,798 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,798 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:02,798 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:02,798 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:02,799 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,799 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:02,801 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:02,801 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:02,801 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:02,802 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:02,815 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:02,815 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:02,815 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:02,815 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:02,815 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:02,817 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,817 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,817 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:02,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,818 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,820 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:02,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:02,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:02,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:02,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:02,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:02,827 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:02,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:02,827 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:02,927 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:02,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:02,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:02,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:02,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:02,930 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:02,930 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:02,930 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:02,930 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:02,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,932 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,932 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:02,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:02,932 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,932 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:02,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:02,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:02,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:02,933 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:02,933 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:02,933 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:02,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:02,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:02,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:02,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:02,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:02,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:02,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:02,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:02,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:02,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:02,936 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:02,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:02,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eac0922[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:02,938 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:02,938 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,939 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:02,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:02,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:02,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:02,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:02,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:02,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:02,946 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:02,946 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:02,946 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:02,946 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:02,946 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:02,947 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,948 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,948 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:02,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:02,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:02,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,954 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:02,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:02,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:02,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,955 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:02,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:02,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:02,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:02,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:02,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:02,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,955 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:02,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:02,955 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:02,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,064 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:03,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:03,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:03,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:03,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:03,068 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:03,068 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,068 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:03,068 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:03,068 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,068 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:03,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:03,071 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,072 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:03,072 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:03,072 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:03,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:03,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:03,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:03,074 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:03,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8b54c8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:03,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:03,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:03,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:03,080 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:03,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:03,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:03,083 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:03,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:03,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:03,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:03,083 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:03,086 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,086 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:03,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:03,087 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,089 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:03,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,091 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:03,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:03,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:03,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:03,091 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:03,091 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:03,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:03,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:55:03,162 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:55:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:55:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:55:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:55:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:03,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720903173, max timestamp=1680720903178Num uniques 859; 2023-04-05 18:55:03,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:55:03,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720903173, max timestamp=1680720903178Num uniques 859; 2023-04-05 18:55:03,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720903173, max timestamp=1680720903178Num uniques 859; 2023-04-05 18:55:03,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:55:03,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:55:03,191 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:03,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:03,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:03,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:03,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:03,195 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:03,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:03,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:03,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720963192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720963193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,196 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,197 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,197 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:03,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:03,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:03,198 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,198 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:03,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,198 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:03,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720963194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,198 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:03,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:03,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720963195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:03,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:03,200 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,200 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,200 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,200 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:03,202 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:03,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a76770e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:03,202 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,203 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:03,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,204 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:03,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:03,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:03,205 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:03,205 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:03,205 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,205 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:03,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:03,207 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:03,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:03,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:03,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:03,207 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:03,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:03,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,210 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,211 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,213 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:03,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:03,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:03,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:03,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:03,213 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:03,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,213 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:03,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:03,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720963297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720963297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720963299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720963300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,313 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:03,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:03,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:03,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:03,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:03,316 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:03,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:03,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:03,317 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,317 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,319 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:03,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:03,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:03,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:03,321 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,321 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:03,321 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:03,321 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:03,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:03,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:03,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:03,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:03,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:03,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@770fe3a7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:03,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:03,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,325 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:03,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:03,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:03,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:03,326 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:03,326 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:03,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,326 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:03,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:03,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:03,329 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:03,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:03,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:03,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:03,329 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:03,336 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,336 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,336 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:03,337 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:03,340 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:03,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:03,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:03,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:03,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:03,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:03,341 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:03,341 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:03,441 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:03,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:03,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:03,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:03,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:03,444 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:03,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:03,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:03,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,446 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:03,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:03,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:03,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:03,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:03,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:03,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:03,448 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,448 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:03,448 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:03,448 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:03,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:03,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:03,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:03,449 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,449 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:03,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f4017d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:03,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:03,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,453 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:03,453 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,454 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:03,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:03,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:03,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:03,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:03,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:03,457 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:03,457 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:03,457 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:03,457 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:03,457 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:03,459 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,459 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,459 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:03,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:03,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:03,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:03,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:03,467 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:03,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,467 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:03,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:03,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720963498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720963500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720963502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720963503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,568 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:03,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:03,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:03,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:03,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/39ae77404d6f444cba9dd8a997f57151 2023-04-05 18:55:03,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:03,571 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:03,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:03,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:03,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,572 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:03,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:03,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:03,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:03,575 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,575 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:03,575 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:03,575 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:03,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:03,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e1e259f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:03,579 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:03,579 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,580 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:03,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:03,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:03,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/16a728f5ac354595b70036ea29bab46f 2023-04-05 18:55:03,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:03,586 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:03,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:03,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:03,590 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:03,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:03,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:03,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:03,590 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:03,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/ecebe124e9024a3fa28ca508af4f49a6 2023-04-05 18:55:03,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:03,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:03,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:03,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:03,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:03,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:03,593 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:03,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,595 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:03,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/39ae77404d6f444cba9dd8a997f57151 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/39ae77404d6f444cba9dd8a997f57151 2023-04-05 18:55:03,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:03,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:03,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,598 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:03,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:03,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:03,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:03,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:03,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,598 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:03,598 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:03,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:03,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/39ae77404d6f444cba9dd8a997f57151, entries=150, sequenceid=672, filesize=11.9 K 2023-04-05 18:55:03,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/16a728f5ac354595b70036ea29bab46f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/16a728f5ac354595b70036ea29bab46f 2023-04-05 18:55:03,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/16a728f5ac354595b70036ea29bab46f, entries=150, sequenceid=672, filesize=11.9 K 2023-04-05 18:55:03,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/ecebe124e9024a3fa28ca508af4f49a6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/ecebe124e9024a3fa28ca508af4f49a6 2023-04-05 18:55:03,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/ecebe124e9024a3fa28ca508af4f49a6, entries=150, sequenceid=672, filesize=11.9 K 2023-04-05 18:55:03,609 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for c2dced816e56beff0d38901aab0de290 in 447ms, sequenceid=672, compaction requested=true 2023-04-05 18:55:03,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:03,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:03,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:03,610 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:03,610 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:03,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:03,611 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52138 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:03,611 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:03,611 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/A is initiating minor compaction (all files) 2023-04-05 18:55:03,611 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/B is initiating minor compaction (all files) 2023-04-05 18:55:03,611 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/A in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,611 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/B in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,611 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/497014520cdf4619a201203735a5b771, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/7cf388e79bb4420588ac9b67a363d3d1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/b60b4cf6c364439d8572eb1d0d7fa188, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/39ae77404d6f444cba9dd8a997f57151] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=50.9 K 2023-04-05 18:55:03,611 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/813c48cc885d42dd91f400a204085ec6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/963e083961494254b5638e3484d18777, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5e24b53d3d574693aacc22f0136fbaa4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/16a728f5ac354595b70036ea29bab46f] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=48.5 K 2023-04-05 18:55:03,612 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 497014520cdf4619a201203735a5b771, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680720900061 2023-04-05 18:55:03,612 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 813c48cc885d42dd91f400a204085ec6, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680720900061 2023-04-05 18:55:03,612 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7cf388e79bb4420588ac9b67a363d3d1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1680720900377 2023-04-05 18:55:03,612 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 963e083961494254b5638e3484d18777, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1680720900377 2023-04-05 18:55:03,612 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting b60b4cf6c364439d8572eb1d0d7fa188, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1680720901036 2023-04-05 18:55:03,612 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 5e24b53d3d574693aacc22f0136fbaa4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1680720901036 2023-04-05 18:55:03,613 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 39ae77404d6f444cba9dd8a997f57151, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1680720903157 2023-04-05 18:55:03,613 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 16a728f5ac354595b70036ea29bab46f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1680720903157 2023-04-05 18:55:03,620 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#A#compaction#293 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:03,622 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#B#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:03,698 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:03,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:03,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:03,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:03,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:03,701 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:03,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:03,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:03,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:03,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:03,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:03,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:03,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:03,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:03,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:03,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:03,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:03,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:03,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:03,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:03,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:03,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:03,705 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-05 18:55:03,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:55:03,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:03,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:55:03,706 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:03,706 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:55:03,706 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:03,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:55:03,805 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:03,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720963810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720963810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720963811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720963813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720963914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720963914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720963914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:03,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:03,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720963916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,030 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/573585afd7a14603840eca08f1af9c28 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/573585afd7a14603840eca08f1af9c28 2023-04-05 18:55:04,031 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/665efb62be134b23bad0cbb4bdd9dc8a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/665efb62be134b23bad0cbb4bdd9dc8a 2023-04-05 18:55:04,035 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2dced816e56beff0d38901aab0de290/A of c2dced816e56beff0d38901aab0de290 into 573585afd7a14603840eca08f1af9c28(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:04,035 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:04,035 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/A, priority=12, startTime=1680720903610; duration=0sec 2023-04-05 18:55:04,035 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:04,035 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:04,036 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2dced816e56beff0d38901aab0de290/B of c2dced816e56beff0d38901aab0de290 into 665efb62be134b23bad0cbb4bdd9dc8a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:04,036 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:04,036 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:04,036 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/B, priority=12, startTime=1680720903610; duration=0sec 2023-04-05 18:55:04,036 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): c2dced816e56beff0d38901aab0de290/C is initiating minor compaction (all files) 2023-04-05 18:55:04,037 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c2dced816e56beff0d38901aab0de290/C in TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:04,037 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:04,037 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/80baae30296d4731a5ba7f10bd9137cf, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/95f40f7880714fd8bd6afdf2d302f8f6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/56d761f16605438994996e365ea93aac, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/ecebe124e9024a3fa28ca508af4f49a6] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp, totalSize=48.5 K 2023-04-05 18:55:04,037 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 80baae30296d4731a5ba7f10bd9137cf, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680720900061 2023-04-05 18:55:04,037 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 95f40f7880714fd8bd6afdf2d302f8f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1680720900377 2023-04-05 18:55:04,038 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 56d761f16605438994996e365ea93aac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1680720901036 2023-04-05 18:55:04,038 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting ecebe124e9024a3fa28ca508af4f49a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1680720903157 2023-04-05 18:55:04,045 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): c2dced816e56beff0d38901aab0de290#C#compaction#296 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:04,053 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/26d5967b1c92439e848b12045b64a0e5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/26d5967b1c92439e848b12045b64a0e5 2023-04-05 18:55:04,056 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c2dced816e56beff0d38901aab0de290/C of c2dced816e56beff0d38901aab0de290 into 26d5967b1c92439e848b12045b64a0e5(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:04,056 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:04,056 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290., storeName=c2dced816e56beff0d38901aab0de290/C, priority=12, startTime=1680720903610; duration=0sec 2023-04-05 18:55:04,057 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:04,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:04,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34500 deadline: 1680720964116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:04,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34510 deadline: 1680720964116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:04,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34522 deadline: 1680720964118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,121 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/8f02b7d105f640beae7d09321d459fef 2023-04-05 18:55:04,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:04,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34570 deadline: 1680720964120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,131 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/55692f60b0244152bb6a7340ca315bfc 2023-04-05 18:55:04,147 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/c802c642e3544da3bcdf07d5a41d4503 2023-04-05 18:55:04,151 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/8f02b7d105f640beae7d09321d459fef as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8f02b7d105f640beae7d09321d459fef 2023-04-05 18:55:04,155 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8f02b7d105f640beae7d09321d459fef, entries=150, sequenceid=708, filesize=11.9 K 2023-04-05 18:55:04,156 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/55692f60b0244152bb6a7340ca315bfc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/55692f60b0244152bb6a7340ca315bfc 2023-04-05 18:55:04,160 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/55692f60b0244152bb6a7340ca315bfc, entries=150, sequenceid=708, filesize=11.9 K 2023-04-05 18:55:04,161 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/c802c642e3544da3bcdf07d5a41d4503 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/c802c642e3544da3bcdf07d5a41d4503 2023-04-05 18:55:04,164 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/c802c642e3544da3bcdf07d5a41d4503, entries=150, sequenceid=708, filesize=11.9 K 2023-04-05 18:55:04,165 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for c2dced816e56beff0d38901aab0de290 in 460ms, sequenceid=708, compaction requested=false 2023-04-05 18:55:04,165 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:04,165 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:04,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:04,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:04,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,168 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:04,168 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,170 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,171 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:04,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:04,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7789e955[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:04,172 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:04,172 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:04,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:04,175 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:04,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:04,178 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:04,178 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:04,178 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:04,178 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:04,178 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:04,180 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,180 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:04,180 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:04,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:04,181 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:04,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:04,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:04,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:04,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:04,186 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:04,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:04,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:04,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:04,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:04,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,186 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:04,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:04,186 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:04,287 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:04,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:04,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:04,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:04,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:04,290 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:04,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:04,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:04,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:04,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,290 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:04,291 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:04,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:04,292 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:04,292 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:04,292 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:04,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:04,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:04,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:04,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:04,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:04,293 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:04,294 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:04,294 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:04,294 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:04,294 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:04,294 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:04,294 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. started... 2023-04-05 18:55:04,294 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 18:55:04,294 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:55:04,294 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:04,294 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:55:04,294 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:04,294 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:55:04,294 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:04,303 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=720 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/55dc79e9d0bb47b8ae0df31eb4a98824 2023-04-05 18:55:04,312 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=720 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/55a580f4e1db4f73817a6c3a903c7c97 2023-04-05 18:55:04,325 DEBUG [Thread-1605] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:49755 2023-04-05 18:55:04,325 DEBUG [Thread-1605] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:04,326 DEBUG [Thread-1601] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:49755 2023-04-05 18:55:04,326 DEBUG [Thread-1601] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:04,327 DEBUG [Thread-1603] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:49755 2023-04-05 18:55:04,327 DEBUG [Thread-1609] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54a39be8 to 127.0.0.1:49755 2023-04-05 18:55:04,327 DEBUG [Thread-1603] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:04,327 DEBUG [Thread-1609] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:04,328 DEBUG [Thread-1607] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de064cb to 127.0.0.1:49755 2023-04-05 18:55:04,328 DEBUG [Thread-1607] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:04,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on c2dced816e56beff0d38901aab0de290 2023-04-05 18:55:04,419 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. as already flushing 2023-04-05 18:55:04,419 DEBUG [Thread-1590] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:49755 2023-04-05 18:55:04,419 DEBUG [Thread-1590] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:04,422 DEBUG [Thread-1592] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:49755 2023-04-05 18:55:04,422 DEBUG [Thread-1598] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08cd7cb0 to 127.0.0.1:49755 2023-04-05 18:55:04,422 DEBUG [Thread-1592] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:04,422 DEBUG [Thread-1598] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:04,424 DEBUG [Thread-1596] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7810a231 to 127.0.0.1:49755 2023-04-05 18:55:04,424 DEBUG [Thread-1596] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:04,720 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=720 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/8b3835dc190d4c90bca690e388ea5dac 2023-04-05 18:55:04,723 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/55dc79e9d0bb47b8ae0df31eb4a98824 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/55dc79e9d0bb47b8ae0df31eb4a98824 2023-04-05 18:55:04,726 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/55dc79e9d0bb47b8ae0df31eb4a98824, entries=150, sequenceid=720, filesize=11.9 K 2023-04-05 18:55:04,727 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/55a580f4e1db4f73817a6c3a903c7c97 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/55a580f4e1db4f73817a6c3a903c7c97 2023-04-05 18:55:04,731 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/55a580f4e1db4f73817a6c3a903c7c97, entries=150, sequenceid=720, filesize=11.9 K 2023-04-05 18:55:04,732 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/8b3835dc190d4c90bca690e388ea5dac as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/8b3835dc190d4c90bca690e388ea5dac 2023-04-05 18:55:04,735 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/8b3835dc190d4c90bca690e388ea5dac, entries=150, sequenceid=720, filesize=11.9 K 2023-04-05 18:55:04,735 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=26.84 KB/27480 for c2dced816e56beff0d38901aab0de290 in 441ms, sequenceid=720, compaction requested=true 2023-04-05 18:55:04,735 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:04,735 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:04,735 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:04,735 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:04,735 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,736 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:04,736 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:04,737 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,737 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:04,737 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,737 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f132ed8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:04,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:04,739 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:04,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:04,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:04,740 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,740 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:04,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:04,742 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:04,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:04,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:04,744 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:04,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:04,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:04,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:04,744 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:04,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:04,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:04,746 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,747 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:04,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:04,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:04,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:04,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:04,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:04,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:04,751 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:04,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:04,751 DEBUG [Thread-1600] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:04,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:04,751 DEBUG [Thread-1600] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:04,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:04,752 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:04,752 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:04,852 DEBUG [Thread-1600] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:04,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:06,208 DEBUG [Thread-1594] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:49755 2023-04-05 18:55:06,208 DEBUG [Thread-1594] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:06,208 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 18:55:06,208 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 169 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 115 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 171 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 124 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 5305 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 5209 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 5278 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 5328 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 5204 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 18:55:06,209 INFO [Listener at localhost/43705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 18:55:06,209 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:49755 2023-04-05 18:55:06,209 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:06,210 INFO [Listener at localhost/43705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 18:55:06,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 18:55:06,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:06,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 18:55:06,213 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720906213"}]},"ts":"1680720906213"} 2023-04-05 18:55:06,214 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 18:55:06,215 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 18:55:06,216 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2dced816e56beff0d38901aab0de290, UNASSIGN}] 2023-04-05 18:55:06,217 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2dced816e56beff0d38901aab0de290, UNASSIGN 2023-04-05 18:55:06,217 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=c2dced816e56beff0d38901aab0de290, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:06,217 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720906217"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720906217"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720906217"}]},"ts":"1680720906217"} 2023-04-05 18:55:06,218 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:55:06,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 18:55:06,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c2dced816e56beff0d38901aab0de290 2023-04-05 18:55:06,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c2dced816e56beff0d38901aab0de290, disabling compactions & flushes 2023-04-05 18:55:06,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:06,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:06,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. after waiting 0 ms 2023-04-05 18:55:06,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:06,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c2dced816e56beff0d38901aab0de290 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 18:55:06,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=A 2023-04-05 18:55:06,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:06,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=B 2023-04-05 18:55:06,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:06,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c2dced816e56beff0d38901aab0de290, store=C 2023-04-05 18:55:06,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:06,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 18:55:06,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/242dcc18eda14af5835f10baadaee19f 2023-04-05 18:55:06,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 18:55:07,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/e086b1a025e443f993a89560796a9f10 2023-04-05 18:55:07,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 18:55:07,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/7a6641349027464bbf8b468ed3734a62 2023-04-05 18:55:07,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/A/242dcc18eda14af5835f10baadaee19f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/242dcc18eda14af5835f10baadaee19f 2023-04-05 18:55:07,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/242dcc18eda14af5835f10baadaee19f, entries=100, sequenceid=728, filesize=9.5 K 2023-04-05 18:55:07,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/B/e086b1a025e443f993a89560796a9f10 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/e086b1a025e443f993a89560796a9f10 2023-04-05 18:55:07,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/e086b1a025e443f993a89560796a9f10, entries=100, sequenceid=728, filesize=9.5 K 2023-04-05 18:55:07,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/.tmp/C/7a6641349027464bbf8b468ed3734a62 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7a6641349027464bbf8b468ed3734a62 2023-04-05 18:55:07,610 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7a6641349027464bbf8b468ed3734a62, entries=100, sequenceid=728, filesize=9.5 K 2023-04-05 18:55:07,610 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for c2dced816e56beff0d38901aab0de290 in 1240ms, sequenceid=728, compaction requested=true 2023-04-05 18:55:07,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/6e753e9f73c948fc8a4e5c8fcbd8504e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8d22f68e40054443a61e9ddf060dc1cc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/1bfcec3172ff4ddaa542f535415fceb2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/f7403edb09034f1c9819e4e29720d412, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/43101259b5da4beca76d0baccabf003b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9bf6408917344eb282b18d50df85d6af, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/267d9890552c442aa78a4f5791a25098, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c98539fd2de4418faaa0b10dbd735df5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9dde5f43e93844929895ae586e33f0f6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/2ad042cbc18f4f00b9ee5873cf8ffee2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9807f17e20f4e38926d203787fcfbe6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/22450ea399164d01a55ee39720f5fbf4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/aba1447cad3a40b7be0713904f627cc7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/644c9dfad96940a1939d4a14bacc4bb8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/e7b95641801549278c5979c39da69b71, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8b33f98b5d20491bbc8180d53c627b33, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8a4bbc25f7b24687ab766fbaa7a3ee4e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/da1d12d0695743b6b48e90bebe1a3d25, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/edca4b38c9634ba4af269a572d6d3f65, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/492e2a746e3a47c7a4b617c1a4f53028, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/d4ab5e8c7c084f988b26254cf74f0049, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8857acc64c3c4e448b622571759ecc43, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/66ed109f1b92465ba0464ad435dc60a0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df1e401e19ed474680446ba9fd88d7ab, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9cf625d712e04a4ca51a59eb5e685cdf, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9bfd8b4333f46978e5f39e86d42fb95, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df14e3d744124ef8b19e96940949ceca, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/315ecf69713445ea90788c71e3c74e34, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/497014520cdf4619a201203735a5b771, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/7cf388e79bb4420588ac9b67a363d3d1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/b60b4cf6c364439d8572eb1d0d7fa188, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/39ae77404d6f444cba9dd8a997f57151] to archive 2023-04-05 18:55:07,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:55:07,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/6e753e9f73c948fc8a4e5c8fcbd8504e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/6e753e9f73c948fc8a4e5c8fcbd8504e 2023-04-05 18:55:07,637 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8d22f68e40054443a61e9ddf060dc1cc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8d22f68e40054443a61e9ddf060dc1cc 2023-04-05 18:55:07,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/1bfcec3172ff4ddaa542f535415fceb2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/1bfcec3172ff4ddaa542f535415fceb2 2023-04-05 18:55:07,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/f7403edb09034f1c9819e4e29720d412 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/f7403edb09034f1c9819e4e29720d412 2023-04-05 18:55:07,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/43101259b5da4beca76d0baccabf003b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/43101259b5da4beca76d0baccabf003b 2023-04-05 18:55:07,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9bf6408917344eb282b18d50df85d6af to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9bf6408917344eb282b18d50df85d6af 2023-04-05 18:55:07,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/267d9890552c442aa78a4f5791a25098 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/267d9890552c442aa78a4f5791a25098 2023-04-05 18:55:07,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c98539fd2de4418faaa0b10dbd735df5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c98539fd2de4418faaa0b10dbd735df5 2023-04-05 18:55:07,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9dde5f43e93844929895ae586e33f0f6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9dde5f43e93844929895ae586e33f0f6 2023-04-05 18:55:07,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/2ad042cbc18f4f00b9ee5873cf8ffee2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/2ad042cbc18f4f00b9ee5873cf8ffee2 2023-04-05 18:55:07,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9807f17e20f4e38926d203787fcfbe6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9807f17e20f4e38926d203787fcfbe6 2023-04-05 18:55:07,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/22450ea399164d01a55ee39720f5fbf4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/22450ea399164d01a55ee39720f5fbf4 2023-04-05 18:55:07,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/aba1447cad3a40b7be0713904f627cc7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/aba1447cad3a40b7be0713904f627cc7 2023-04-05 18:55:07,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/644c9dfad96940a1939d4a14bacc4bb8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/644c9dfad96940a1939d4a14bacc4bb8 2023-04-05 18:55:07,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/e7b95641801549278c5979c39da69b71 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/e7b95641801549278c5979c39da69b71 2023-04-05 18:55:07,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8b33f98b5d20491bbc8180d53c627b33 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8b33f98b5d20491bbc8180d53c627b33 2023-04-05 18:55:07,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8a4bbc25f7b24687ab766fbaa7a3ee4e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8a4bbc25f7b24687ab766fbaa7a3ee4e 2023-04-05 18:55:07,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/da1d12d0695743b6b48e90bebe1a3d25 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/da1d12d0695743b6b48e90bebe1a3d25 2023-04-05 18:55:07,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/edca4b38c9634ba4af269a572d6d3f65 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/edca4b38c9634ba4af269a572d6d3f65 2023-04-05 18:55:07,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/492e2a746e3a47c7a4b617c1a4f53028 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/492e2a746e3a47c7a4b617c1a4f53028 2023-04-05 18:55:07,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/d4ab5e8c7c084f988b26254cf74f0049 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/d4ab5e8c7c084f988b26254cf74f0049 2023-04-05 18:55:07,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8857acc64c3c4e448b622571759ecc43 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8857acc64c3c4e448b622571759ecc43 2023-04-05 18:55:07,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/66ed109f1b92465ba0464ad435dc60a0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/66ed109f1b92465ba0464ad435dc60a0 2023-04-05 18:55:07,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df1e401e19ed474680446ba9fd88d7ab to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df1e401e19ed474680446ba9fd88d7ab 2023-04-05 18:55:07,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9cf625d712e04a4ca51a59eb5e685cdf to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/9cf625d712e04a4ca51a59eb5e685cdf 2023-04-05 18:55:07,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9bfd8b4333f46978e5f39e86d42fb95 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/c9bfd8b4333f46978e5f39e86d42fb95 2023-04-05 18:55:07,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df14e3d744124ef8b19e96940949ceca to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/df14e3d744124ef8b19e96940949ceca 2023-04-05 18:55:07,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/315ecf69713445ea90788c71e3c74e34 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/315ecf69713445ea90788c71e3c74e34 2023-04-05 18:55:07,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/497014520cdf4619a201203735a5b771 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/497014520cdf4619a201203735a5b771 2023-04-05 18:55:07,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/7cf388e79bb4420588ac9b67a363d3d1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/7cf388e79bb4420588ac9b67a363d3d1 2023-04-05 18:55:07,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/b60b4cf6c364439d8572eb1d0d7fa188 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/b60b4cf6c364439d8572eb1d0d7fa188 2023-04-05 18:55:07,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/39ae77404d6f444cba9dd8a997f57151 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/39ae77404d6f444cba9dd8a997f57151 2023-04-05 18:55:07,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/4784481d7c1342e78a3b953e40f351b9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/03b51afbf0834e1daa3891f1877f36a4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b34d0c5d65b4ee4a882372f58477c47, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/0c98276ef3d94c84a7d04d77cf7de601, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c5968215bfd44a87ac1ec0db1e21978d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2337ad528a1143509adf5c626ebd3ce4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c897ffa0d673456a846989e056d87184, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/71eef36d4e164de881948ca48dedfe48, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/0a1ba0c8feff4740a25cb117e199e134, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8745866b265f4703b44c7f00af26aeb2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c49c2efefa0b417bb8f5978306daddc5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/a589866cc61640fe974cc2574a99e899, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/adc455bd936740729edc7d85ecebb33e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/1631a6e438724c36bda0a8539a070c4f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/20c52f6e41b947b2a8aa862d3bf5e8ec, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/207696d261b34f74879f41c70277b55e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/895b27a065d247b3a8f5a93aa65e116a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/3d7cf6a6e004425088a98ec7a48ad8ac, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/13ec9af0258244909fe4843a3d6099f7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/6172690017134516924e7fd1633d2da5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b3a2159664245f6b0100be1e75e39dc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8219d4430cc64bafb18158247e937f05, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/dcc76613ba2a435c987ea05689117de7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bb79d19d4c0643ca8ce0498a78713583, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/92052549788c4f66807aceef8a751a17, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5c50569f54874db7ac15bb4f5fdf8098, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bae561a1931748d2947fcb2ac426548e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/813c48cc885d42dd91f400a204085ec6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/331fe60caa0342509e06cfd109aa39f9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/963e083961494254b5638e3484d18777, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5e24b53d3d574693aacc22f0136fbaa4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/16a728f5ac354595b70036ea29bab46f] to archive 2023-04-05 18:55:07,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:55:07,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/4784481d7c1342e78a3b953e40f351b9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/4784481d7c1342e78a3b953e40f351b9 2023-04-05 18:55:07,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/03b51afbf0834e1daa3891f1877f36a4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/03b51afbf0834e1daa3891f1877f36a4 2023-04-05 18:55:07,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b34d0c5d65b4ee4a882372f58477c47 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b34d0c5d65b4ee4a882372f58477c47 2023-04-05 18:55:07,673 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/0c98276ef3d94c84a7d04d77cf7de601 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/0c98276ef3d94c84a7d04d77cf7de601 2023-04-05 18:55:07,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c5968215bfd44a87ac1ec0db1e21978d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c5968215bfd44a87ac1ec0db1e21978d 2023-04-05 18:55:07,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2337ad528a1143509adf5c626ebd3ce4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2337ad528a1143509adf5c626ebd3ce4 2023-04-05 18:55:07,675 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c897ffa0d673456a846989e056d87184 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c897ffa0d673456a846989e056d87184 2023-04-05 18:55:07,676 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/71eef36d4e164de881948ca48dedfe48 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/71eef36d4e164de881948ca48dedfe48 2023-04-05 18:55:07,677 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/0a1ba0c8feff4740a25cb117e199e134 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/0a1ba0c8feff4740a25cb117e199e134 2023-04-05 18:55:07,678 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8745866b265f4703b44c7f00af26aeb2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8745866b265f4703b44c7f00af26aeb2 2023-04-05 18:55:07,679 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c49c2efefa0b417bb8f5978306daddc5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/c49c2efefa0b417bb8f5978306daddc5 2023-04-05 18:55:07,680 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/a589866cc61640fe974cc2574a99e899 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/a589866cc61640fe974cc2574a99e899 2023-04-05 18:55:07,681 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/adc455bd936740729edc7d85ecebb33e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/adc455bd936740729edc7d85ecebb33e 2023-04-05 18:55:07,682 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/1631a6e438724c36bda0a8539a070c4f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/1631a6e438724c36bda0a8539a070c4f 2023-04-05 18:55:07,683 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/20c52f6e41b947b2a8aa862d3bf5e8ec to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/20c52f6e41b947b2a8aa862d3bf5e8ec 2023-04-05 18:55:07,683 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/207696d261b34f74879f41c70277b55e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/207696d261b34f74879f41c70277b55e 2023-04-05 18:55:07,684 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/895b27a065d247b3a8f5a93aa65e116a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/895b27a065d247b3a8f5a93aa65e116a 2023-04-05 18:55:07,685 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/3d7cf6a6e004425088a98ec7a48ad8ac to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/3d7cf6a6e004425088a98ec7a48ad8ac 2023-04-05 18:55:07,686 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/13ec9af0258244909fe4843a3d6099f7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/13ec9af0258244909fe4843a3d6099f7 2023-04-05 18:55:07,687 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/6172690017134516924e7fd1633d2da5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/6172690017134516924e7fd1633d2da5 2023-04-05 18:55:07,688 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b3a2159664245f6b0100be1e75e39dc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/2b3a2159664245f6b0100be1e75e39dc 2023-04-05 18:55:07,688 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8219d4430cc64bafb18158247e937f05 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/8219d4430cc64bafb18158247e937f05 2023-04-05 18:55:07,689 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/dcc76613ba2a435c987ea05689117de7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/dcc76613ba2a435c987ea05689117de7 2023-04-05 18:55:07,690 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bb79d19d4c0643ca8ce0498a78713583 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bb79d19d4c0643ca8ce0498a78713583 2023-04-05 18:55:07,691 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/92052549788c4f66807aceef8a751a17 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/92052549788c4f66807aceef8a751a17 2023-04-05 18:55:07,692 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5c50569f54874db7ac15bb4f5fdf8098 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5c50569f54874db7ac15bb4f5fdf8098 2023-04-05 18:55:07,692 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bae561a1931748d2947fcb2ac426548e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/bae561a1931748d2947fcb2ac426548e 2023-04-05 18:55:07,693 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/813c48cc885d42dd91f400a204085ec6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/813c48cc885d42dd91f400a204085ec6 2023-04-05 18:55:07,694 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/331fe60caa0342509e06cfd109aa39f9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/331fe60caa0342509e06cfd109aa39f9 2023-04-05 18:55:07,695 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/963e083961494254b5638e3484d18777 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/963e083961494254b5638e3484d18777 2023-04-05 18:55:07,696 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5e24b53d3d574693aacc22f0136fbaa4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/5e24b53d3d574693aacc22f0136fbaa4 2023-04-05 18:55:07,697 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/16a728f5ac354595b70036ea29bab46f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/16a728f5ac354595b70036ea29bab46f 2023-04-05 18:55:07,699 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1c28af11683e498ab83f8a4a6630ec43, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/c553296f927f41dcbecb4f3e31c2542b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7b8f12fb65c44ffd8b38738352e19abb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/db6f707b14a94da0b32dc8b9f2ed2ef1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/f10f6bf9aa5840949b43f324f0e66eb6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/dc6b0ed18b4b434cb937d02ea664b4aa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/6dd137d21b964616bd50013d84b41274, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/16cb460b9a1d40d3865f09df04f9865e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7f7a3fd1922d40328177114a304b95ed, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/67725fc7df7643fbaca5a313129de54c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/2fabf7552fb74a6a9828e5b2848742e2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e9cc990c62554eb9a6fe47e889b4f419, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7bb779bfa1d945649cc5ceefa0185bbc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/8485464c88b445849cc107916f9c74d4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bb7a1a4277f74d9592abc81c1c13539d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/03c4f2728b0f4bc7b39915471b0bc698, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/279ad0c2a68f4336bad299636a20d45c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1ea4f0c4ecc24eedaf70e3cc21e56ee9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bed199d3eac7458585391100a7b9e5be, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7484e9580eb74b4890a0b17cc09257d3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/47dbd76c2f3841c294238fa11a8f3e4b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1fa363bee5d6484e81e5bc297f4398e8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/eda1e8d629ae44b9b54b5ca783854f33, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/46fea136538c453ea22a9806093fdfac, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/3c67746109d0483a82252c583f857126, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/d8ae542492214dd79293eb519d25d918, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/9080d679280d4421ab5c50ee920c6c80, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/80baae30296d4731a5ba7f10bd9137cf, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e7c3b05b3df3470597e6c785d70ac539, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/95f40f7880714fd8bd6afdf2d302f8f6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/56d761f16605438994996e365ea93aac, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/ecebe124e9024a3fa28ca508af4f49a6] to archive 2023-04-05 18:55:07,700 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:55:07,701 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1c28af11683e498ab83f8a4a6630ec43 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1c28af11683e498ab83f8a4a6630ec43 2023-04-05 18:55:07,702 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/c553296f927f41dcbecb4f3e31c2542b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/c553296f927f41dcbecb4f3e31c2542b 2023-04-05 18:55:07,703 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7b8f12fb65c44ffd8b38738352e19abb to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7b8f12fb65c44ffd8b38738352e19abb 2023-04-05 18:55:07,704 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/db6f707b14a94da0b32dc8b9f2ed2ef1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/db6f707b14a94da0b32dc8b9f2ed2ef1 2023-04-05 18:55:07,705 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/f10f6bf9aa5840949b43f324f0e66eb6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/f10f6bf9aa5840949b43f324f0e66eb6 2023-04-05 18:55:07,706 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/dc6b0ed18b4b434cb937d02ea664b4aa to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/dc6b0ed18b4b434cb937d02ea664b4aa 2023-04-05 18:55:07,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/6dd137d21b964616bd50013d84b41274 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/6dd137d21b964616bd50013d84b41274 2023-04-05 18:55:07,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/16cb460b9a1d40d3865f09df04f9865e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/16cb460b9a1d40d3865f09df04f9865e 2023-04-05 18:55:07,708 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7f7a3fd1922d40328177114a304b95ed to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7f7a3fd1922d40328177114a304b95ed 2023-04-05 18:55:07,709 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/67725fc7df7643fbaca5a313129de54c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/67725fc7df7643fbaca5a313129de54c 2023-04-05 18:55:07,710 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/2fabf7552fb74a6a9828e5b2848742e2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/2fabf7552fb74a6a9828e5b2848742e2 2023-04-05 18:55:07,710 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e9cc990c62554eb9a6fe47e889b4f419 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e9cc990c62554eb9a6fe47e889b4f419 2023-04-05 18:55:07,711 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7bb779bfa1d945649cc5ceefa0185bbc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7bb779bfa1d945649cc5ceefa0185bbc 2023-04-05 18:55:07,712 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/8485464c88b445849cc107916f9c74d4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/8485464c88b445849cc107916f9c74d4 2023-04-05 18:55:07,713 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bb7a1a4277f74d9592abc81c1c13539d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bb7a1a4277f74d9592abc81c1c13539d 2023-04-05 18:55:07,714 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/03c4f2728b0f4bc7b39915471b0bc698 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/03c4f2728b0f4bc7b39915471b0bc698 2023-04-05 18:55:07,715 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/279ad0c2a68f4336bad299636a20d45c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/279ad0c2a68f4336bad299636a20d45c 2023-04-05 18:55:07,716 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1ea4f0c4ecc24eedaf70e3cc21e56ee9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1ea4f0c4ecc24eedaf70e3cc21e56ee9 2023-04-05 18:55:07,717 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bed199d3eac7458585391100a7b9e5be to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/bed199d3eac7458585391100a7b9e5be 2023-04-05 18:55:07,718 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7484e9580eb74b4890a0b17cc09257d3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7484e9580eb74b4890a0b17cc09257d3 2023-04-05 18:55:07,719 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/47dbd76c2f3841c294238fa11a8f3e4b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/47dbd76c2f3841c294238fa11a8f3e4b 2023-04-05 18:55:07,719 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1fa363bee5d6484e81e5bc297f4398e8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/1fa363bee5d6484e81e5bc297f4398e8 2023-04-05 18:55:07,720 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/eda1e8d629ae44b9b54b5ca783854f33 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/eda1e8d629ae44b9b54b5ca783854f33 2023-04-05 18:55:07,721 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/46fea136538c453ea22a9806093fdfac to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/46fea136538c453ea22a9806093fdfac 2023-04-05 18:55:07,722 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/3c67746109d0483a82252c583f857126 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/3c67746109d0483a82252c583f857126 2023-04-05 18:55:07,723 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/d8ae542492214dd79293eb519d25d918 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/d8ae542492214dd79293eb519d25d918 2023-04-05 18:55:07,724 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/9080d679280d4421ab5c50ee920c6c80 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/9080d679280d4421ab5c50ee920c6c80 2023-04-05 18:55:07,725 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/80baae30296d4731a5ba7f10bd9137cf to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/80baae30296d4731a5ba7f10bd9137cf 2023-04-05 18:55:07,725 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e7c3b05b3df3470597e6c785d70ac539 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/e7c3b05b3df3470597e6c785d70ac539 2023-04-05 18:55:07,726 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/95f40f7880714fd8bd6afdf2d302f8f6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/95f40f7880714fd8bd6afdf2d302f8f6 2023-04-05 18:55:07,727 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/56d761f16605438994996e365ea93aac to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/56d761f16605438994996e365ea93aac 2023-04-05 18:55:07,728 DEBUG [StoreCloser-TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/ecebe124e9024a3fa28ca508af4f49a6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/ecebe124e9024a3fa28ca508af4f49a6 2023-04-05 18:55:07,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/recovered.edits/731.seqid, newMaxSeqId=731, maxSeqId=1 2023-04-05 18:55:07,733 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290. 2023-04-05 18:55:07,733 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c2dced816e56beff0d38901aab0de290: 2023-04-05 18:55:07,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c2dced816e56beff0d38901aab0de290 2023-04-05 18:55:07,734 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=c2dced816e56beff0d38901aab0de290, regionState=CLOSED 2023-04-05 18:55:07,735 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720907734"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720907734"}]},"ts":"1680720907734"} 2023-04-05 18:55:07,737 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-05 18:55:07,737 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure c2dced816e56beff0d38901aab0de290, server=jenkins-hbase4.apache.org,33489,1680720816595 in 1.5170 sec 2023-04-05 18:55:07,738 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-05 18:55:07,738 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c2dced816e56beff0d38901aab0de290, UNASSIGN in 1.5220 sec 2023-04-05 18:55:07,738 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720907738"}]},"ts":"1680720907738"} 2023-04-05 18:55:07,739 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 18:55:07,742 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 18:55:07,743 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5320 sec 2023-04-05 18:55:08,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 18:55:08,317 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-05 18:55:08,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 18:55:08,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:08,319 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:08,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-05 18:55:08,320 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:08,323 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290 2023-04-05 18:55:08,325 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/recovered.edits] 2023-04-05 18:55:08,328 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/242dcc18eda14af5835f10baadaee19f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/242dcc18eda14af5835f10baadaee19f 2023-04-05 18:55:08,329 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/55dc79e9d0bb47b8ae0df31eb4a98824 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/55dc79e9d0bb47b8ae0df31eb4a98824 2023-04-05 18:55:08,330 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/573585afd7a14603840eca08f1af9c28 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/573585afd7a14603840eca08f1af9c28 2023-04-05 18:55:08,331 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8f02b7d105f640beae7d09321d459fef to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/A/8f02b7d105f640beae7d09321d459fef 2023-04-05 18:55:08,333 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/55692f60b0244152bb6a7340ca315bfc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/55692f60b0244152bb6a7340ca315bfc 2023-04-05 18:55:08,334 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/55a580f4e1db4f73817a6c3a903c7c97 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/55a580f4e1db4f73817a6c3a903c7c97 2023-04-05 18:55:08,335 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/665efb62be134b23bad0cbb4bdd9dc8a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/665efb62be134b23bad0cbb4bdd9dc8a 2023-04-05 18:55:08,336 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/e086b1a025e443f993a89560796a9f10 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/B/e086b1a025e443f993a89560796a9f10 2023-04-05 18:55:08,338 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/26d5967b1c92439e848b12045b64a0e5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/26d5967b1c92439e848b12045b64a0e5 2023-04-05 18:55:08,339 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7a6641349027464bbf8b468ed3734a62 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/7a6641349027464bbf8b468ed3734a62 2023-04-05 18:55:08,340 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/8b3835dc190d4c90bca690e388ea5dac to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/8b3835dc190d4c90bca690e388ea5dac 2023-04-05 18:55:08,341 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/c802c642e3544da3bcdf07d5a41d4503 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/C/c802c642e3544da3bcdf07d5a41d4503 2023-04-05 18:55:08,343 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/recovered.edits/731.seqid to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290/recovered.edits/731.seqid 2023-04-05 18:55:08,343 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/c2dced816e56beff0d38901aab0de290 2023-04-05 18:55:08,343 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:55:08,345 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:08,347 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 18:55:08,348 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 18:55:08,349 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:08,349 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 18:55:08,349 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680720908349"}]},"ts":"9223372036854775807"} 2023-04-05 18:55:08,350 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 18:55:08,350 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c2dced816e56beff0d38901aab0de290, NAME => 'TestAcidGuarantees,,1680720882140.c2dced816e56beff0d38901aab0de290.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 18:55:08,350 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 18:55:08,350 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680720908350"}]},"ts":"9223372036854775807"} 2023-04-05 18:55:08,352 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 18:55:08,353 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:08,354 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 36 msec 2023-04-05 18:55:08,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-05 18:55:08,421 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-05 18:55:08,432 INFO [Listener at localhost/43705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 225) - Thread LEAK? -, OpenFileDescriptor=555 (was 537) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=413 (was 360) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1299 (was 1418) 2023-04-05 18:55:08,442 INFO [Listener at localhost/43705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=555, MaxFileDescriptor=60000, SystemLoadAverage=413, ProcessCount=169, AvailableMemoryMB=1299 2023-04-05 18:55:08,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 18:55:08,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 18:55:08,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:08,446 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 18:55:08,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-05 18:55:08,446 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 18:55:08,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 18:55:08,447 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:08,448 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d empty. 2023-04-05 18:55:08,448 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:08,448 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:55:08,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 18:55:08,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 18:55:08,854 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 18:55:08,856 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 64f9b0847f90f48ecb0b352322029c8d, NAME => 'TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp 2023-04-05 18:55:08,862 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:55:08,862 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 64f9b0847f90f48ecb0b352322029c8d, disabling compactions & flushes 2023-04-05 18:55:08,862 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:08,862 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:08,862 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. after waiting 0 ms 2023-04-05 18:55:08,862 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:08,862 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:08,862 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:08,864 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 18:55:08,865 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720908865"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720908865"}]},"ts":"1680720908865"} 2023-04-05 18:55:08,866 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 18:55:08,867 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 18:55:08,867 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720908867"}]},"ts":"1680720908867"} 2023-04-05 18:55:08,868 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 18:55:08,870 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, ASSIGN}] 2023-04-05 18:55:08,872 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, ASSIGN 2023-04-05 18:55:08,872 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33489,1680720816595; forceNewPlan=false, retain=false 2023-04-05 18:55:09,023 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=64f9b0847f90f48ecb0b352322029c8d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:09,024 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720909023"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720909023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720909023"}]},"ts":"1680720909023"} 2023-04-05 18:55:09,025 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:55:09,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 18:55:09,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:09,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 64f9b0847f90f48ecb0b352322029c8d, NAME => 'TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:55:09,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:09,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:55:09,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:09,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:09,181 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:09,183 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A 2023-04-05 18:55:09,183 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A 2023-04-05 18:55:09,183 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:55:09,183 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64f9b0847f90f48ecb0b352322029c8d columnFamilyName A 2023-04-05 18:55:09,184 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(310): Store=64f9b0847f90f48ecb0b352322029c8d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:55:09,184 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:09,185 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B 2023-04-05 18:55:09,185 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B 2023-04-05 18:55:09,185 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:55:09,186 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64f9b0847f90f48ecb0b352322029c8d columnFamilyName B 2023-04-05 18:55:09,186 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(310): Store=64f9b0847f90f48ecb0b352322029c8d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:55:09,186 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:09,187 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C 2023-04-05 18:55:09,187 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C 2023-04-05 18:55:09,188 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:55:09,188 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64f9b0847f90f48ecb0b352322029c8d columnFamilyName C 2023-04-05 18:55:09,189 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(310): Store=64f9b0847f90f48ecb0b352322029c8d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:55:09,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:09,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:09,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:09,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:55:09,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:09,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 18:55:09,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 64f9b0847f90f48ecb0b352322029c8d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60665850, jitterRate=-0.09600839018821716}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:55:09,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:09,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., pid=37, masterSystemTime=1680720909176 2023-04-05 18:55:09,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:09,197 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:09,198 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=64f9b0847f90f48ecb0b352322029c8d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:09,198 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720909198"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680720909198"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680720909198"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680720909198"}]},"ts":"1680720909198"} 2023-04-05 18:55:09,200 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-05 18:55:09,200 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 in 174 msec 2023-04-05 18:55:09,201 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-05 18:55:09,201 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, ASSIGN in 330 msec 2023-04-05 18:55:09,202 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 18:55:09,202 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720909202"}]},"ts":"1680720909202"} 2023-04-05 18:55:09,203 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 18:55:09,205 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 18:55:09,206 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 761 msec 2023-04-05 18:55:09,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 18:55:09,550 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-05 18:55:09,551 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:09,556 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:09,557 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:09,559 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39942, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:09,560 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 18:55:09,561 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52930, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 18:55:09,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 18:55:09,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 18:55:09,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:09,976 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 18:55:09,977 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 18:55:09,977 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 18:55:09,979 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-05 18:55:09,982 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, REOPEN/MOVE}] 2023-04-05 18:55:09,983 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, REOPEN/MOVE 2023-04-05 18:55:09,983 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=64f9b0847f90f48ecb0b352322029c8d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:09,983 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720909983"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720909983"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720909983"}]},"ts":"1680720909983"} 2023-04-05 18:55:09,984 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:55:10,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 64f9b0847f90f48ecb0b352322029c8d, disabling compactions & flushes 2023-04-05 18:55:10,137 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:10,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:10,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. after waiting 0 ms 2023-04-05 18:55:10,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:10,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-05 18:55:10,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:10,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:10,153 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 64f9b0847f90f48ecb0b352322029c8d to self. 2023-04-05 18:55:10,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,154 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=64f9b0847f90f48ecb0b352322029c8d, regionState=CLOSED 2023-04-05 18:55:10,155 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720910154"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720910154"}]},"ts":"1680720910154"} 2023-04-05 18:55:10,156 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-05 18:55:10,157 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 in 171 msec 2023-04-05 18:55:10,157 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,33489,1680720816595; forceNewPlan=false, retain=true 2023-04-05 18:55:10,307 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=64f9b0847f90f48ecb0b352322029c8d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,308 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720910307"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720910307"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720910307"}]},"ts":"1680720910307"} 2023-04-05 18:55:10,309 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:55:10,464 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:10,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 64f9b0847f90f48ecb0b352322029c8d, NAME => 'TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:55:10,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:55:10,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,466 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,467 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A 2023-04-05 18:55:10,467 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A 2023-04-05 18:55:10,467 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:55:10,467 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64f9b0847f90f48ecb0b352322029c8d columnFamilyName A 2023-04-05 18:55:10,469 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(310): Store=64f9b0847f90f48ecb0b352322029c8d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:55:10,469 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,470 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B 2023-04-05 18:55:10,470 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B 2023-04-05 18:55:10,470 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:55:10,470 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64f9b0847f90f48ecb0b352322029c8d columnFamilyName B 2023-04-05 18:55:10,471 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(310): Store=64f9b0847f90f48ecb0b352322029c8d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:55:10,471 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,472 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C 2023-04-05 18:55:10,472 DEBUG [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C 2023-04-05 18:55:10,472 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:55:10,472 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64f9b0847f90f48ecb0b352322029c8d columnFamilyName C 2023-04-05 18:55:10,473 INFO [StoreOpener-64f9b0847f90f48ecb0b352322029c8d-1] regionserver.HStore(310): Store=64f9b0847f90f48ecb0b352322029c8d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:55:10,473 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:10,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,474 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,476 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:55:10,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,478 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 64f9b0847f90f48ecb0b352322029c8d; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60752939, jitterRate=-0.09471066296100616}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:55:10,478 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:10,478 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., pid=42, masterSystemTime=1680720910460 2023-04-05 18:55:10,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:10,480 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:10,480 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=64f9b0847f90f48ecb0b352322029c8d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,480 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720910480"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680720910480"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680720910480"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680720910480"}]},"ts":"1680720910480"} 2023-04-05 18:55:10,482 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-05 18:55:10,482 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 in 172 msec 2023-04-05 18:55:10,484 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-05 18:55:10,484 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, REOPEN/MOVE in 500 msec 2023-04-05 18:55:10,486 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-05 18:55:10,486 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 505 msec 2023-04-05 18:55:10,488 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 923 msec 2023-04-05 18:55:10,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-05 18:55:10,489 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:10,494 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:10,495 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:10,507 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:10,507 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:10,511 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:10,511 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x06a0df34 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:10,514 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe1db6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:10,514 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:10,518 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:10,519 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:10,522 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:10,522 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:10,525 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:10,525 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:10,530 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:10,530 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x0eb8f142 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:10,535 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e079da1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:10,535 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x5ce79633 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:10,538 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@159c5e0c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:10,544 DEBUG [hconnection-0x20da9911-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:10,544 DEBUG [hconnection-0x1b9c6994-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:10,544 DEBUG [hconnection-0x7975222e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:10,545 DEBUG [hconnection-0xc3f75cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:10,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:10,546 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39952, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:10,546 DEBUG [hconnection-0x69dee453-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:10,546 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39966, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:10,546 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39986, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:10,547 DEBUG [hconnection-0x53009f8c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:10,547 DEBUG [hconnection-0xc9bf9d8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:10,547 DEBUG [hconnection-0x72b51559-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:10,547 DEBUG [hconnection-0xf4dceec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:10,549 DEBUG [hconnection-0x4f52a090-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:10,550 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:10,550 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40000, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:10,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:10,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:10,551 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:10,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:10,551 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40034, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:10,552 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:10,553 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:10,553 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:10,554 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39982, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:10,554 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40002, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:10,554 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:10,556 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:10,556 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:10,561 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,561 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:10,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:10,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:10,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:10,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:10,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:10,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:10,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:10,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:10,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:10,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:10,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:10,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:10,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:10,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:10,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:10,567 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:10,567 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:10,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:10,570 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:55:10,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:10,572 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:10,572 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:10,572 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:10,572 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:10,572 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:10,572 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:10,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720970590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720970591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720970596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720970599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720970599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,612 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405fb67ac22c7934a489e1c365e3a90e06a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405fb67ac22c7934a489e1c365e3a90e06a 2023-04-05 18:55:10,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720970700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720970700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720970702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720970702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720970702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720970905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720970905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720970907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720970908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:10,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:10,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720970910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,032 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/1fba84a183884af788c5aef63a9b02e7 2023-04-05 18:55:11,042 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/0c61fc8ed4b14a98b3a4e990b2170e51 2023-04-05 18:55:11,050 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/68ac5559e9ae48c3a957bdedd2657c20 2023-04-05 18:55:11,055 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/1fba84a183884af788c5aef63a9b02e7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/1fba84a183884af788c5aef63a9b02e7 2023-04-05 18:55:11,059 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/1fba84a183884af788c5aef63a9b02e7, entries=150, sequenceid=17, filesize=25.1 K 2023-04-05 18:55:11,061 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/0c61fc8ed4b14a98b3a4e990b2170e51 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/0c61fc8ed4b14a98b3a4e990b2170e51 2023-04-05 18:55:11,065 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/0c61fc8ed4b14a98b3a4e990b2170e51, entries=150, sequenceid=17, filesize=11.6 K 2023-04-05 18:55:11,065 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/68ac5559e9ae48c3a957bdedd2657c20 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/68ac5559e9ae48c3a957bdedd2657c20 2023-04-05 18:55:11,068 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/68ac5559e9ae48c3a957bdedd2657c20, entries=150, sequenceid=17, filesize=11.6 K 2023-04-05 18:55:11,069 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 64f9b0847f90f48ecb0b352322029c8d in 499ms, sequenceid=17, compaction requested=false 2023-04-05 18:55:11,069 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-05 18:55:11,071 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:11,071 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:11,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:11,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:11,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:11,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:11,073 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,073 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:11,073 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:11,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:11,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:11,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a527ae[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:11,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:11,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:11,077 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:11,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:11,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:11,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:11,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:11,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:11,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:11,079 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:11,079 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:11,079 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:11,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:11,081 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:11,081 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:11,081 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:11,081 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:11,081 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:11,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:11,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:11,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:11,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:11,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:11,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,085 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:11,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:11,088 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:11,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:11,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:11,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:11,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:11,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:11,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:11,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:11,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:11,088 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:11,088 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:11,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:11,188 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:11,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:11,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:11,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:11,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:11,192 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:11,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:11,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:11,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:11,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:11,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:11,198 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,198 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:11,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:11,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:11,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:11,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:11,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:11,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:11,200 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:11,200 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:11,200 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:11,201 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:11,201 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:11,201 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:55:11,201 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:11,201 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:11,202 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:11,202 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:11,202 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:11,202 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:11,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:11,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:11,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720971218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720971219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720971220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720971220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720971221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720971324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720971324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720971324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720971325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720971326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720971527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720971527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720971528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720971528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720971530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,616 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405f2a1f168ae844676bbcf4b3052bde00a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f2a1f168ae844676bbcf4b3052bde00a 2023-04-05 18:55:11,621 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/9db6447a941e424badeae9aa7934b5c8 2023-04-05 18:55:11,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720971832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720971832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720971832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720971834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:11,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:11,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720971835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,031 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/30198d6b2be94408ab362aa03cbbff16 2023-04-05 18:55:12,043 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/83562c1ff5ea47df8d287f6745f19b3f 2023-04-05 18:55:12,046 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/9db6447a941e424badeae9aa7934b5c8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/9db6447a941e424badeae9aa7934b5c8 2023-04-05 18:55:12,050 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/9db6447a941e424badeae9aa7934b5c8, entries=150, sequenceid=37, filesize=25.1 K 2023-04-05 18:55:12,051 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/30198d6b2be94408ab362aa03cbbff16 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/30198d6b2be94408ab362aa03cbbff16 2023-04-05 18:55:12,057 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/30198d6b2be94408ab362aa03cbbff16, entries=150, sequenceid=37, filesize=11.6 K 2023-04-05 18:55:12,058 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/83562c1ff5ea47df8d287f6745f19b3f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/83562c1ff5ea47df8d287f6745f19b3f 2023-04-05 18:55:12,077 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/83562c1ff5ea47df8d287f6745f19b3f, entries=150, sequenceid=37, filesize=11.6 K 2023-04-05 18:55:12,078 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 64f9b0847f90f48ecb0b352322029c8d in 877ms, sequenceid=37, compaction requested=false 2023-04-05 18:55:12,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:12,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:12,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:12,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:12,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:12,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:12,080 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:12,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:12,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6948de38[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:12,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:12,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:12,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:12,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:12,085 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,085 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:12,086 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:12,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:12,089 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:12,089 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:12,089 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:12,089 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:12,089 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:12,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:12,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:12,091 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:12,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:12,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:12,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:12,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:12,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:12,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:12,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:12,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:12,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:12,102 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:12,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:12,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:12,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:12,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:12,103 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:12,103 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:12,203 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:12,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:12,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:12,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:12,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:12,207 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:12,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:12,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:12,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:12,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:12,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:12,209 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:12,209 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:12,209 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,209 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:12,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:12,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:12,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:12,211 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:12,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:12,341 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:12,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680720912352, max timestamp=1680720912380Num uniques 859; 2023-04-05 18:55:12,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:12,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680720912352, max timestamp=1680720912380Num uniques 859; 2023-04-05 18:55:12,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680720912352, max timestamp=1680720912380Num uniques 859; 2023-04-05 18:55:12,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:12,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:12,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720972406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720972407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720972414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720972414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720972415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720972519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720972519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720972522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720972522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720972523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,624 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405b6c3053a9ddf4a7d897f7cf258b82897 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b6c3053a9ddf4a7d897f7cf258b82897 2023-04-05 18:55:12,643 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/f859841fedc84747b2c468d08f344fbb 2023-04-05 18:55:12,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720972726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720972726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720972726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720972727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:12,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:12,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720972728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720973033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720973033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720973033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720973034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720973034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,064 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/97500f9cda8848f28a4e6b4d9a1f7b07 2023-04-05 18:55:13,265 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 18:55:13,476 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/77714b42ab874ae8a54c763f10a4453d 2023-04-05 18:55:13,480 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/f859841fedc84747b2c468d08f344fbb as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/f859841fedc84747b2c468d08f344fbb 2023-04-05 18:55:13,484 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/f859841fedc84747b2c468d08f344fbb, entries=150, sequenceid=49, filesize=25.1 K 2023-04-05 18:55:13,486 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/97500f9cda8848f28a4e6b4d9a1f7b07 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/97500f9cda8848f28a4e6b4d9a1f7b07 2023-04-05 18:55:13,489 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/97500f9cda8848f28a4e6b4d9a1f7b07, entries=150, sequenceid=49, filesize=11.6 K 2023-04-05 18:55:13,490 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/77714b42ab874ae8a54c763f10a4453d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77714b42ab874ae8a54c763f10a4453d 2023-04-05 18:55:13,497 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77714b42ab874ae8a54c763f10a4453d, entries=150, sequenceid=49, filesize=11.6 K 2023-04-05 18:55:13,498 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 64f9b0847f90f48ecb0b352322029c8d in 1287ms, sequenceid=49, compaction requested=true 2023-04-05 18:55:13,498 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:13,498 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:13,498 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:13,498 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:13,498 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,498 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:13,498 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,500 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,501 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,501 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51b35105[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:13,502 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:13,502 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,503 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:13,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:13,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:13,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:13,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:13,505 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:13,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:13,509 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:13,509 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:13,510 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:13,510 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:13,510 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:13,512 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,512 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:13,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,512 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,512 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:13,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:13,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:13,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:13,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:13,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:13,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:13,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:13,516 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:13,516 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:13,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:13,544 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-05 18:55:13,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:13,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:13,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:13,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:13,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:13,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:13,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720973551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720973551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720973556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720973557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720973559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,616 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:13,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:13,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:13,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:13,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:13,620 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:13,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:13,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:13,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:13,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,622 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:13,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:13,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:13,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:13,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:13,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:13,625 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:13,625 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:13,625 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:13,625 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:13,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:13,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:13,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:13,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:13,626 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:13,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37b0d005[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:13,633 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:13,633 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,635 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:13,635 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,635 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:13,637 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:13,637 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:13,637 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:13,637 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:13,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:13,642 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:13,642 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:13,642 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:13,642 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:13,642 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:13,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:13,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:13,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:13,645 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,647 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,649 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:13,649 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:13,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:13,649 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:13,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:13,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:13,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:13,650 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:13,650 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:13,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720973660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720973661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720973663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720973665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720973668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,750 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:13,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:13,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:13,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:13,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:13,768 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:13,768 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:13,768 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:13,768 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:13,769 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,769 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,772 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:13,772 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:13,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:13,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:13,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:13,781 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:13,781 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:13,781 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:13,781 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:13,781 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:13,781 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:13,781 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:13,781 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:13,781 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,781 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:13,781 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:13,782 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,782 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:13,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28b7190a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:13,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:13,787 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:13,788 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,788 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:13,790 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:13,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:13,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:13,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:13,795 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:13,795 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:13,795 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:13,795 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:13,795 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:13,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,799 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:13,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,799 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,801 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,804 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:13,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:13,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:13,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:13,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:13,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:13,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:13,804 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:13,804 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:13,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:13,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720973865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720973866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720973872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720973872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:13,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720973880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,904 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:13,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:13,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:13,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:13,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:13,920 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:13,920 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:13,920 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:13,920 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:13,921 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,921 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,925 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:13,925 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:13,925 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,925 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:13,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:13,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:13,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:13,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:13,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:13,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:13,940 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:13,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:13,940 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:13,940 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:13,940 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:13,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:13,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:13,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:13,940 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:13,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@721ab3a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:13,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:13,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,948 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:13,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:13,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:13,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:13,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:13,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:13,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:13,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:13,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:13,953 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:13,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:13,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:13,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:13,954 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:13,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:13,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:13,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,962 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,963 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304054e9757b26f204d90b1ce84ad52ccae17 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054e9757b26f204d90b1ce84ad52ccae17 2023-04-05 18:55:13,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:13,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:13,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:13,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:13,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:13,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:13,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:13,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:13,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:13,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:13,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:13,968 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:13,968 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:13,986 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/38e71152e6874b17b022907afbc008e5 2023-04-05 18:55:14,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/69d50d3cee4d4492a0404f420dd7fd01 2023-04-05 18:55:14,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/f4ac4e2d39ee437e806edb71af5e406f 2023-04-05 18:55:14,055 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/38e71152e6874b17b022907afbc008e5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/38e71152e6874b17b022907afbc008e5 2023-04-05 18:55:14,059 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/38e71152e6874b17b022907afbc008e5, entries=200, sequenceid=84, filesize=31.9 K 2023-04-05 18:55:14,060 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/69d50d3cee4d4492a0404f420dd7fd01 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/69d50d3cee4d4492a0404f420dd7fd01 2023-04-05 18:55:14,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/69d50d3cee4d4492a0404f420dd7fd01, entries=150, sequenceid=84, filesize=11.6 K 2023-04-05 18:55:14,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/f4ac4e2d39ee437e806edb71af5e406f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f4ac4e2d39ee437e806edb71af5e406f 2023-04-05 18:55:14,068 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:14,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:14,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f4ac4e2d39ee437e806edb71af5e406f, entries=150, sequenceid=84, filesize=11.6 K 2023-04-05 18:55:14,070 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 64f9b0847f90f48ecb0b352322029c8d in 526ms, sequenceid=84, compaction requested=true 2023-04-05 18:55:14,070 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:14,071 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:14,071 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:14,071 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:14,071 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:14,071 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:14,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:14,073 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:14,073 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/A is initiating minor compaction (all files) 2023-04-05 18:55:14,073 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/A in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:14,073 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/1fba84a183884af788c5aef63a9b02e7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/9db6447a941e424badeae9aa7934b5c8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/f859841fedc84747b2c468d08f344fbb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/38e71152e6874b17b022907afbc008e5] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=107.3 K 2023-04-05 18:55:14,074 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:14,074 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/B is initiating minor compaction (all files) 2023-04-05 18:55:14,074 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/B in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:14,074 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/0c61fc8ed4b14a98b3a4e990b2170e51, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/30198d6b2be94408ab362aa03cbbff16, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/97500f9cda8848f28a4e6b4d9a1f7b07, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/69d50d3cee4d4492a0404f420dd7fd01] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=46.4 K 2023-04-05 18:55:14,076 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 1fba84a183884af788c5aef63a9b02e7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680720910563 2023-04-05 18:55:14,076 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0c61fc8ed4b14a98b3a4e990b2170e51, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680720910563 2023-04-05 18:55:14,076 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 9db6447a941e424badeae9aa7934b5c8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680720910587 2023-04-05 18:55:14,076 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 30198d6b2be94408ab362aa03cbbff16, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680720910587 2023-04-05 18:55:14,077 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting f859841fedc84747b2c468d08f344fbb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720911218 2023-04-05 18:55:14,077 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 97500f9cda8848f28a4e6b4d9a1f7b07, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720911218 2023-04-05 18:55:14,077 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 38e71152e6874b17b022907afbc008e5, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680720912405 2023-04-05 18:55:14,077 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 69d50d3cee4d4492a0404f420dd7fd01, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680720912413 2023-04-05 18:55:14,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:14,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:14,092 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:14,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:14,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:14,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:14,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:14,094 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:14,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:14,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:14,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:14,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:14,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:14,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:14,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:14,104 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:14,104 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:14,104 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:14,110 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#B#compaction#318 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:14,111 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:14,111 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:14,111 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:14,111 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:14,112 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:55:14,112 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#A#compaction#317 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:14,112 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:14,112 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:14,112 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:14,112 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:14,112 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:14,112 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:14,159 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d69a867a2d5b4d44b3bd24900fafc3a2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d69a867a2d5b4d44b3bd24900fafc3a2 2023-04-05 18:55:14,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:14,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:14,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720974211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680720914198, max timestamp=1680720914214Num uniques 859; 2023-04-05 18:55:14,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720914201, max timestamp=1680720914214Num uniques 859; 2023-04-05 18:55:14,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-05 18:55:14,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:14,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720914201, max timestamp=1680720914214Num uniques 859; 2023-04-05 18:55:14,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:14,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720974252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720974255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720974255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720974256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720974316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720974357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720974365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720974366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720974366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720974523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,541 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/7dda715ad7824fa4a7dac7515ec9368d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/7dda715ad7824fa4a7dac7515ec9368d 2023-04-05 18:55:14,549 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/B of 64f9b0847f90f48ecb0b352322029c8d into 7dda715ad7824fa4a7dac7515ec9368d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:14,549 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:14,549 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/B, priority=12, startTime=1680720914071; duration=0sec 2023-04-05 18:55:14,549 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:14,549 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:14,551 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:14,551 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/C is initiating minor compaction (all files) 2023-04-05 18:55:14,551 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/C in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:14,551 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/68ac5559e9ae48c3a957bdedd2657c20, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/83562c1ff5ea47df8d287f6745f19b3f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77714b42ab874ae8a54c763f10a4453d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f4ac4e2d39ee437e806edb71af5e406f] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=46.4 K 2023-04-05 18:55:14,552 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 68ac5559e9ae48c3a957bdedd2657c20, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680720910563 2023-04-05 18:55:14,552 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 83562c1ff5ea47df8d287f6745f19b3f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680720910587 2023-04-05 18:55:14,556 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 77714b42ab874ae8a54c763f10a4453d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720911218 2023-04-05 18:55:14,557 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting f4ac4e2d39ee437e806edb71af5e406f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680720912413 2023-04-05 18:55:14,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720974567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,570 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#C#compaction#320 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:14,571 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=95, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/29318a358f04496fa9c4938fed7c47d0 2023-04-05 18:55:14,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720974572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720974573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720974574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,620 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/c517fdd8522545efb15be77d6034f599 2023-04-05 18:55:14,650 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/c056ceb0a53644e983cd2fcb754a5a11 2023-04-05 18:55:14,657 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/29318a358f04496fa9c4938fed7c47d0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/29318a358f04496fa9c4938fed7c47d0 2023-04-05 18:55:14,663 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/29318a358f04496fa9c4938fed7c47d0, entries=150, sequenceid=95, filesize=25.1 K 2023-04-05 18:55:14,663 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/c517fdd8522545efb15be77d6034f599 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/c517fdd8522545efb15be77d6034f599 2023-04-05 18:55:14,667 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/c517fdd8522545efb15be77d6034f599, entries=150, sequenceid=95, filesize=11.6 K 2023-04-05 18:55:14,668 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/c056ceb0a53644e983cd2fcb754a5a11 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c056ceb0a53644e983cd2fcb754a5a11 2023-04-05 18:55:14,674 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c056ceb0a53644e983cd2fcb754a5a11, entries=150, sequenceid=95, filesize=11.6 K 2023-04-05 18:55:14,675 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 64f9b0847f90f48ecb0b352322029c8d in 563ms, sequenceid=95, compaction requested=false 2023-04-05 18:55:14,675 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:14,675 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:14,675 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:14,675 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:14,675 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,675 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:14,675 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:14,685 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:14,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:14,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd1f9a2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:14,687 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:14,687 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:14,688 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:14,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:14,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:14,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:14,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:14,689 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,689 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:14,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:14,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:14,690 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:14,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:14,693 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:14,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:14,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:14,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:14,693 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:14,695 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,695 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:14,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,695 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:14,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:14,695 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:14,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,699 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:14,699 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:14,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:14,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:14,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:14,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:14,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:14,699 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:14,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:14,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:14,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,700 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:14,700 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:14,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:14,800 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:14,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:14,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:14,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:14,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:14,803 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:14,804 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:14,804 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:14,804 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:14,804 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,804 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:14,805 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,805 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:14,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:14,805 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,805 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:14,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:14,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:14,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:14,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:14,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:14,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:14,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:14,807 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:14,807 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:14,807 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:14,807 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:14,807 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:14,807 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:14,807 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:14,807 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=129.71 KB heapSize=365.45 KB 2023-04-05 18:55:14,807 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:14,807 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:14,808 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:14,808 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:14,808 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:14,808 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:14,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:14,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:14,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720974872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720974872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720974883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720974883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720974883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:14,967 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/4f49e8e6b9794c7680bec77f06107aa8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/4f49e8e6b9794c7680bec77f06107aa8 2023-04-05 18:55:14,971 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/A of 64f9b0847f90f48ecb0b352322029c8d into 4f49e8e6b9794c7680bec77f06107aa8(size=25.3 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:14,971 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:14,971 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/A, priority=12, startTime=1680720914071; duration=0sec 2023-04-05 18:55:14,972 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:14,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:14,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720974980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:15,012 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/008b7b5b46e741d1be78fad663e5b6fe as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/008b7b5b46e741d1be78fad663e5b6fe 2023-04-05 18:55:15,017 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/C of 64f9b0847f90f48ecb0b352322029c8d into 008b7b5b46e741d1be78fad663e5b6fe(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:15,017 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:15,017 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/C, priority=12, startTime=1680720914071; duration=0sec 2023-04-05 18:55:15,017 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:15,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:15,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720975187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:15,229 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d6d182c10a2b4a869e2b8ea6505eb436 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d6d182c10a2b4a869e2b8ea6505eb436 2023-04-05 18:55:15,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:15,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720975382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:15,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:15,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720975393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:15,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:15,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720975393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:15,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:15,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720975393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:15,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:15,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720975492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:15,638 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=131, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/21d238494e16403caf06a3dfd0bb7ecb 2023-04-05 18:55:16,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:16,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720975998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,047 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/fe569ac2825e4e54979573db391d1af1 2023-04-05 18:55:16,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:16,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720976392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720976405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:16,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720976410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:16,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720976411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,457 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/288490062f8a467e8b633aab9ae0ee2f 2023-04-05 18:55:16,462 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/21d238494e16403caf06a3dfd0bb7ecb as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/21d238494e16403caf06a3dfd0bb7ecb 2023-04-05 18:55:16,466 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/21d238494e16403caf06a3dfd0bb7ecb, entries=150, sequenceid=131, filesize=25.2 K 2023-04-05 18:55:16,468 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/fe569ac2825e4e54979573db391d1af1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/fe569ac2825e4e54979573db391d1af1 2023-04-05 18:55:16,483 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/fe569ac2825e4e54979573db391d1af1, entries=150, sequenceid=131, filesize=11.7 K 2023-04-05 18:55:16,484 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/288490062f8a467e8b633aab9ae0ee2f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/288490062f8a467e8b633aab9ae0ee2f 2023-04-05 18:55:16,488 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/288490062f8a467e8b633aab9ae0ee2f, entries=150, sequenceid=131, filesize=11.7 K 2023-04-05 18:55:16,489 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~365.41 KB/374176, currentSize=53.67 KB/54960 for 64f9b0847f90f48ecb0b352322029c8d in 1682ms, sequenceid=131, compaction requested=true 2023-04-05 18:55:16,489 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:16,489 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:16,491 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:16,491 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:16,492 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:16,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:16,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:16,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:16,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d5038c5[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:16,494 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:16,495 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:16,496 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:16,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:16,496 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:16,496 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:16,496 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:16,496 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,496 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:16,497 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,497 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:16,497 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:16,497 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:16,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:16,503 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:16,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:16,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:16,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:16,503 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:16,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:16,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:16,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:16,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:16,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:16,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:16,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:16,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:16,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:16,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:16,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,511 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:16,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:16,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,514 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:16,514 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:16,514 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:16,514 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:16,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:16,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:16,514 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:16,515 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,515 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:16,515 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:16,515 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:16,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:16,515 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:16,515 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:16,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:16,615 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:16,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:16,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:16,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:16,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:16,619 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:16,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:16,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:16,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:16,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:16,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:16,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:16,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:16,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:16,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:16,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:16,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:16,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:16,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:16,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:16,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:16,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:16,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:16,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:16,622 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:16,622 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:16,623 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:55:16,623 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:16,623 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:16,623 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:16,623 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:16,623 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:16,623 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:17,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:17,017 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:17,046 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405e232b55315124e3d9b233a76512e4da4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e232b55315124e3d9b233a76512e4da4 2023-04-05 18:55:17,055 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/8d03101121c049128d8e9ea0b67cf1a9 2023-04-05 18:55:17,065 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/cbfb99f3f7a24f719ba88a39a01719d8 2023-04-05 18:55:17,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720917085, max timestamp=1680720917126Num uniques 859; 2023-04-05 18:55:17,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720917085, max timestamp=1680720917126Num uniques 859; 2023-04-05 18:55:17,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:17,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:17,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720917085, max timestamp=1680720917126Num uniques 859; 2023-04-05 18:55:17,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:17,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:17,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720977221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:17,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720977333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,481 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/5712a0fa5d764ec1a8d5224abcd0b4b7 2023-04-05 18:55:17,485 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/8d03101121c049128d8e9ea0b67cf1a9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/8d03101121c049128d8e9ea0b67cf1a9 2023-04-05 18:55:17,488 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/8d03101121c049128d8e9ea0b67cf1a9, entries=150, sequenceid=144, filesize=25.3 K 2023-04-05 18:55:17,489 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/cbfb99f3f7a24f719ba88a39a01719d8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/cbfb99f3f7a24f719ba88a39a01719d8 2023-04-05 18:55:17,492 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/cbfb99f3f7a24f719ba88a39a01719d8, entries=150, sequenceid=144, filesize=11.8 K 2023-04-05 18:55:17,493 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/5712a0fa5d764ec1a8d5224abcd0b4b7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/5712a0fa5d764ec1a8d5224abcd0b4b7 2023-04-05 18:55:17,496 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/5712a0fa5d764ec1a8d5224abcd0b4b7, entries=150, sequenceid=144, filesize=11.8 K 2023-04-05 18:55:17,497 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 64f9b0847f90f48ecb0b352322029c8d in 874ms, sequenceid=144, compaction requested=true 2023-04-05 18:55:17,497 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:17,497 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:17,497 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:17,497 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:17,497 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,497 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:17,497 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:17,501 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,501 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,502 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,502 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:17,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d3f112c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:17,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:17,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,505 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:17,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:17,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:17,505 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,505 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:17,508 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:17,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:17,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:17,511 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:17,511 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:17,511 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:17,511 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:17,511 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:17,512 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,512 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,512 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,513 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:17,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:17,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:17,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:17,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:17,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:17,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:17,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,516 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:17,517 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:17,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,517 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:17,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:17,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-05 18:55:17,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:17,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:17,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:17,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:17,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:17,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:17,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:17,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720977606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,617 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:17,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:17,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:17,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:17,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:17,620 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:17,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:17,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:17,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:17,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,622 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:17,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:17,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:17,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:17,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:17,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:17,623 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:17,623 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:17,623 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:17,623 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:17,623 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:17,623 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:17,623 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:17,624 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:17,624 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:17,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:17,626 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:17,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67cd0fca[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:17,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:17,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:17,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:17,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:17,629 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,629 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:17,630 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:17,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:17,632 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:17,632 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:17,632 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:17,632 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:17,632 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:17,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,634 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,636 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,640 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:17,640 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:17,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:17,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:17,640 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:17,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:17,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:17,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:17,640 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:17,640 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:17,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:17,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720977713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,740 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:17,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:17,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:17,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:17,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:17,743 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:17,743 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:17,743 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:17,743 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:17,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,745 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:17,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:17,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:17,745 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,745 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:17,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:17,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:17,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:17,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:17,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:17,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:17,746 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:17,747 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:17,747 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:17,747 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:17,747 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:17,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:17,748 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,748 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,748 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,748 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:17,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2412f678[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:17,750 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:17,750 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:17,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:17,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:17,751 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,751 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:17,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:17,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,755 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:17,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:17,757 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:17,757 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:17,757 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:17,757 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:17,757 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:17,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:17,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,760 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,762 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,764 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,764 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:17,764 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:17,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:17,764 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,764 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:17,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:17,764 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:17,765 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:17,765 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:17,765 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:17,765 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:17,765 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,765 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,865 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:17,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:17,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:17,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:17,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:17,868 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:17,868 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:17,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:17,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:17,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,870 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:17,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:17,870 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,870 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:17,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:17,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:17,872 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:17,872 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:17,872 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:17,872 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:17,874 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,874 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,875 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,875 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:17,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:17,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f8642f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:17,877 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:17,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:17,882 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,882 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,882 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:17,882 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:17,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:17,884 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:17,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:17,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:17,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:17,885 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:17,887 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:17,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:17,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:17,888 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,889 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,891 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,891 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,891 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:17,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:17,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:17,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:17,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:17,891 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:17,891 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,891 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:17,891 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,891 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:17,892 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,892 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:17,892 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:17,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:17,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720977920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,950 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304051f643721daa6403689a171c917c4b8b4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051f643721daa6403689a171c917c4b8b4 2023-04-05 18:55:17,954 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=181, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/11a2930904e141f8ba50b02702e8c941 2023-04-05 18:55:17,992 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:17,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:17,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:17,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:17,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:17,995 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:17,995 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:17,995 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:17,995 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:17,996 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,996 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,997 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:17,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:17,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:17,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:17,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:17,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:17,998 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:17,998 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:17,998 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:17,999 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:17,999 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:17,999 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:17,999 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:18,000 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,000 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:18,001 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,001 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:18,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:18,002 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:18,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d5bb86d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:18,002 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,004 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:18,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:18,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:18,004 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,004 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:18,005 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:18,005 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:18,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,005 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:18,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:18,007 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:18,008 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:18,008 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:18,008 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:18,008 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:18,009 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,009 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:18,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,009 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:18,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:18,009 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,011 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,012 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:18,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:18,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:18,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:18,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:18,016 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:18,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:18,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:18,017 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:18,017 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:18,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:18,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:18,117 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:18,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:18,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:18,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:18,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:18,121 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:18,121 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:18,121 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:18,121 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:18,123 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,123 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,124 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:18,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:18,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:18,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:18,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:18,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:18,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:18,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:18,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:18,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:18,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:18,127 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,127 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:18,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:18,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42e74d53[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:18,130 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:18,130 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,131 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,131 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:18,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:18,131 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:18,131 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:18,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:18,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:18,137 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:18,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:18,139 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:18,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:18,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:18,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:18,139 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:18,140 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,140 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:18,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:18,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,140 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:18,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:18,141 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,142 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:18,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:18,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:18,144 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:18,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:18,144 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:18,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:18,145 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:18,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,145 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:18,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720978225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,245 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:18,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:18,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:18,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:18,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:18,255 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:18,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:18,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:18,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:18,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:18,258 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:18,258 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,258 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:18,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:18,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:18,259 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:18,259 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:18,259 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:18,260 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:18,260 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:18,260 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:18,260 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:18,261 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:18,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:18,262 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,262 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:18,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:18,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7175c241[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:18,263 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:18,263 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,264 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:18,264 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:18,264 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:18,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:18,266 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:18,266 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:18,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,266 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:18,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:18,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:18,269 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:18,269 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:18,269 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:18,269 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:18,269 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:18,278 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,278 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:18,278 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:18,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,281 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,282 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:18,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:18,287 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:18,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:18,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:18,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:18,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:18,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:18,287 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,288 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:18,288 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:18,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:18,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/a0840ee9feae4d77a49a0cc474c587ce 2023-04-05 18:55:18,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/1f7aa6b1937c488fb711096b8637f225 2023-04-05 18:55:18,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/11a2930904e141f8ba50b02702e8c941 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/11a2930904e141f8ba50b02702e8c941 2023-04-05 18:55:18,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/11a2930904e141f8ba50b02702e8c941, entries=200, sequenceid=181, filesize=32.1 K 2023-04-05 18:55:18,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/a0840ee9feae4d77a49a0cc474c587ce as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0840ee9feae4d77a49a0cc474c587ce 2023-04-05 18:55:18,388 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:18,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:18,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:18,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0840ee9feae4d77a49a0cc474c587ce, entries=150, sequenceid=181, filesize=11.8 K 2023-04-05 18:55:18,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/1f7aa6b1937c488fb711096b8637f225 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1f7aa6b1937c488fb711096b8637f225 2023-04-05 18:55:18,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:18,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:18,396 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:18,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:18,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:18,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:18,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,397 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1f7aa6b1937c488fb711096b8637f225, entries=150, sequenceid=181, filesize=11.8 K 2023-04-05 18:55:18,398 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:18,398 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:18,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 64f9b0847f90f48ecb0b352322029c8d in 860ms, sequenceid=181, compaction requested=true 2023-04-05 18:55:18,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:18,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:18,398 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:55:18,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:18,398 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,398 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:18,398 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:55:18,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:18,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:18,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:18,400 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136208 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:55:18,400 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/A is initiating minor compaction (all files) 2023-04-05 18:55:18,400 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/A in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:18,400 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/4f49e8e6b9794c7680bec77f06107aa8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/29318a358f04496fa9c4938fed7c47d0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/21d238494e16403caf06a3dfd0bb7ecb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/8d03101121c049128d8e9ea0b67cf1a9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/11a2930904e141f8ba50b02702e8c941] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=133.0 K 2023-04-05 18:55:18,400 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:55:18,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:18,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:18,400 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/B is initiating minor compaction (all files) 2023-04-05 18:55:18,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:18,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:18,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:18,400 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/B in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:18,401 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/7dda715ad7824fa4a7dac7515ec9368d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/c517fdd8522545efb15be77d6034f599, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/fe569ac2825e4e54979573db391d1af1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/cbfb99f3f7a24f719ba88a39a01719d8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0840ee9feae4d77a49a0cc474c587ce] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=58.5 K 2023-04-05 18:55:18,401 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:18,401 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:18,401 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:18,401 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 4f49e8e6b9794c7680bec77f06107aa8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680720912413 2023-04-05 18:55:18,401 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:18,401 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 7dda715ad7824fa4a7dac7515ec9368d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680720912413 2023-04-05 18:55:18,401 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:55:18,401 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 29318a358f04496fa9c4938fed7c47d0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680720913549 2023-04-05 18:55:18,401 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:18,401 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:18,401 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting c517fdd8522545efb15be77d6034f599, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680720913549 2023-04-05 18:55:18,402 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:18,402 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:18,402 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting fe569ac2825e4e54979573db391d1af1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680720914253 2023-04-05 18:55:18,402 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:18,402 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 21d238494e16403caf06a3dfd0bb7ecb, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680720914253 2023-04-05 18:55:18,402 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:18,402 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting cbfb99f3f7a24f719ba88a39a01719d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680720914854 2023-04-05 18:55:18,402 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 8d03101121c049128d8e9ea0b67cf1a9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680720914854 2023-04-05 18:55:18,403 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a0840ee9feae4d77a49a0cc474c587ce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680720917204 2023-04-05 18:55:18,403 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 11a2930904e141f8ba50b02702e8c941, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680720917204 2023-04-05 18:55:18,413 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#B#compaction#333 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:18,415 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040576b743bbf82e484cb17be1537a5e48a5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040576b743bbf82e484cb17be1537a5e48a5 2023-04-05 18:55:18,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:18,417 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:18,422 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#A#compaction#334 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:18,439 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/b410e8eb01674ea2b9093c50ace9c805 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b410e8eb01674ea2b9093c50ace9c805 2023-04-05 18:55:18,444 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/A of 64f9b0847f90f48ecb0b352322029c8d into b410e8eb01674ea2b9093c50ace9c805(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:18,444 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:18,444 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/A, priority=11, startTime=1680720918398; duration=0sec 2023-04-05 18:55:18,444 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:18,444 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:55:18,445 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:55:18,445 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/C is initiating minor compaction (all files) 2023-04-05 18:55:18,445 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/C in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:18,446 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/008b7b5b46e741d1be78fad663e5b6fe, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c056ceb0a53644e983cd2fcb754a5a11, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/288490062f8a467e8b633aab9ae0ee2f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/5712a0fa5d764ec1a8d5224abcd0b4b7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1f7aa6b1937c488fb711096b8637f225] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=58.5 K 2023-04-05 18:55:18,446 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 008b7b5b46e741d1be78fad663e5b6fe, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680720912413 2023-04-05 18:55:18,446 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c056ceb0a53644e983cd2fcb754a5a11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680720913549 2023-04-05 18:55:18,446 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 288490062f8a467e8b633aab9ae0ee2f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680720914253 2023-04-05 18:55:18,447 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 5712a0fa5d764ec1a8d5224abcd0b4b7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680720914854 2023-04-05 18:55:18,447 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 1f7aa6b1937c488fb711096b8637f225, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680720917204 2023-04-05 18:55:18,454 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#C#compaction#335 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:18,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680720918450, max timestamp=1680720918461Num uniques 859; 2023-04-05 18:55:18,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:55:18,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680720918450, max timestamp=1680720918461Num uniques 859; 2023-04-05 18:55:18,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:55:18,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680720918450, max timestamp=1680720918461Num uniques 859; 2023-04-05 18:55:18,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:55:18,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720978483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720978489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720978490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720978491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720978592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720978592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720978599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720978599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40034 deadline: 1680720978736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720978799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720978800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720978806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:18,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720978807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:18,822 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/01939b86d0bd410a9be4a84cc51cbd1e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/01939b86d0bd410a9be4a84cc51cbd1e 2023-04-05 18:55:18,827 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=191, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/58ae1fbfa0744a0596b2a4106e04e0bc 2023-04-05 18:55:18,832 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/B of 64f9b0847f90f48ecb0b352322029c8d into 01939b86d0bd410a9be4a84cc51cbd1e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:18,833 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:18,833 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/B, priority=11, startTime=1680720918398; duration=0sec 2023-04-05 18:55:18,833 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:18,846 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/639d40d27b5f4e95b5df805b93b5b114 2023-04-05 18:55:18,865 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/c7bf5f7237d540aeb386f866b76e5f97 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c7bf5f7237d540aeb386f866b76e5f97 2023-04-05 18:55:18,869 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/C of 64f9b0847f90f48ecb0b352322029c8d into c7bf5f7237d540aeb386f866b76e5f97(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:18,870 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:18,870 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/C, priority=11, startTime=1680720918398; duration=0sec 2023-04-05 18:55:18,870 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:19,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720979107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720979108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720979113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720979117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,255 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/005fcd8da3db4267a80a867902e1d5f8 2023-04-05 18:55:19,259 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/58ae1fbfa0744a0596b2a4106e04e0bc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ae1fbfa0744a0596b2a4106e04e0bc 2023-04-05 18:55:19,270 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ae1fbfa0744a0596b2a4106e04e0bc, entries=150, sequenceid=191, filesize=25.3 K 2023-04-05 18:55:19,271 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/639d40d27b5f4e95b5df805b93b5b114 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/639d40d27b5f4e95b5df805b93b5b114 2023-04-05 18:55:19,275 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/639d40d27b5f4e95b5df805b93b5b114, entries=150, sequenceid=191, filesize=11.8 K 2023-04-05 18:55:19,275 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/005fcd8da3db4267a80a867902e1d5f8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/005fcd8da3db4267a80a867902e1d5f8 2023-04-05 18:55:19,279 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/005fcd8da3db4267a80a867902e1d5f8, entries=150, sequenceid=191, filesize=11.8 K 2023-04-05 18:55:19,280 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 64f9b0847f90f48ecb0b352322029c8d in 879ms, sequenceid=191, compaction requested=false 2023-04-05 18:55:19,280 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:19,280 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:19,281 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:19,281 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:19,281 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,281 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:19,281 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:19,282 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:19,283 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,283 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ab7195c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:19,284 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:19,284 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,296 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,296 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:19,296 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:19,296 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:19,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:19,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:19,299 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:19,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:19,302 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:19,302 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:19,302 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:19,302 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:19,302 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:19,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:19,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:19,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:19,308 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:19,309 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:19,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:19,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:19,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:19,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:19,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,313 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:19,313 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:19,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:19,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:19,314 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:19,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:19,413 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:19,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:19,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:19,420 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:19,420 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:19,420 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:19,420 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:19,420 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,420 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:19,423 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:19,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:19,423 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:19,423 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,423 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:19,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:19,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:19,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:19,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:19,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:19,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:19,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:19,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:19,428 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-05 18:55:19,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:19,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:19,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:19,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:19,429 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:19,429 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:19,467 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d965157df3f84a1e86b055dfe7937175 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d965157df3f84a1e86b055dfe7937175 2023-04-05 18:55:19,478 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=229, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/eae144b56cf6435d96d49d21ea0d663f 2023-04-05 18:55:19,509 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/14aa804ce87b4a799613ba9187ea9022 2023-04-05 18:55:19,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:19,620 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:19,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720979626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720979631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720979632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720979632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720979733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720979734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720979739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720979739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40034 deadline: 1680720979753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,930 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/ede5786254f146ec8e01dd01a44b7eca 2023-04-05 18:55:19,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/eae144b56cf6435d96d49d21ea0d663f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/eae144b56cf6435d96d49d21ea0d663f 2023-04-05 18:55:19,938 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/eae144b56cf6435d96d49d21ea0d663f, entries=150, sequenceid=229, filesize=25.3 K 2023-04-05 18:55:19,939 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/14aa804ce87b4a799613ba9187ea9022 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/14aa804ce87b4a799613ba9187ea9022 2023-04-05 18:55:19,943 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/14aa804ce87b4a799613ba9187ea9022, entries=150, sequenceid=229, filesize=11.8 K 2023-04-05 18:55:19,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/ede5786254f146ec8e01dd01a44b7eca as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/ede5786254f146ec8e01dd01a44b7eca 2023-04-05 18:55:19,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720979940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:19,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720979941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,948 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/ede5786254f146ec8e01dd01a44b7eca, entries=150, sequenceid=229, filesize=11.8 K 2023-04-05 18:55:19,948 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 64f9b0847f90f48ecb0b352322029c8d in 520ms, sequenceid=229, compaction requested=true 2023-04-05 18:55:19,948 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:19,948 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,949 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:19,949 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:19,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:19,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:19,952 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:55:19,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:19,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:19,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:19,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:19,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:19,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:19,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:19,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3f4cb7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:19,953 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:19,953 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,954 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:19,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:19,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:19,958 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:19,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:19,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:19,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:19,958 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:19,961 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,961 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:19,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,961 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:19,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:19,962 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,963 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:19,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:19,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:19,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:19,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:19,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:19,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:19,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:19,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:19,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:19,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:19,967 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:19,967 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:19,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:19,981 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304050c31a714c9734f18b24f8bf6b8e4f6eb to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050c31a714c9734f18b24f8bf6b8e4f6eb 2023-04-05 18:55:20,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720980015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720920001, max timestamp=1680720920018Num uniques 859; 2023-04-05 18:55:20,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:20,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720920001, max timestamp=1680720920018Num uniques 859; 2023-04-05 18:55:20,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:20,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720920001, max timestamp=1680720920018Num uniques 859; 2023-04-05 18:55:20,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:20,067 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:20,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:20,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:20,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:20,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:20,070 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:20,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:20,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:20,071 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,071 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:20,073 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:20,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:20,076 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,077 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:20,077 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:20,077 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:20,078 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,079 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,079 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:20,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a0b76[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:20,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:20,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:20,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:20,084 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:20,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:20,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:20,086 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:20,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:20,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:20,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:20,086 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:20,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,090 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,091 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:20,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,095 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:20,095 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:20,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720980131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720980132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,195 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:20,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:20,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:20,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:20,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:20,199 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:20,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:20,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:20,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,200 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:20,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:20,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:20,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:20,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:20,202 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,202 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:20,202 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:20,202 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:20,203 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:20,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d72a6b4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:20,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:20,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,206 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:20,206 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:20,206 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:20,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:20,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,208 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:20,208 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,208 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:20,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:20,212 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:20,212 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:20,212 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:20,212 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:20,212 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:20,214 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,214 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,214 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,215 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,219 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,219 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,219 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,219 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,219 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,219 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,219 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:20,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:20,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:20,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,219 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,220 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:20,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,220 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,220 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:20,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720980240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720980245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720980249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720980252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,320 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:20,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:20,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:20,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:20,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:20,323 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:20,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:20,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:20,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,328 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,328 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,328 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:20,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:20,330 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,330 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:20,330 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:20,330 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:20,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f5d29bf[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:20,333 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:20,333 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:20,334 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,335 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:20,336 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:20,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:20,338 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:20,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:20,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:20,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:20,338 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:20,341 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,341 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,341 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,341 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,342 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:20,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:20,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,345 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:20,345 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:20,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,345 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:20,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,394 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/d8dd451ce8de488697d2f2c67c9bad83 2023-04-05 18:55:20,405 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/26feb215832e4187a706edee2b2cf35b 2023-04-05 18:55:20,445 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:20,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:20,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:20,448 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:20,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:20,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:20,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,450 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,450 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,450 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,450 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:20,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:20,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:20,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:20,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f078617[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:20,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:20,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,457 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,457 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:20,458 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:20,458 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,458 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:20,458 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:20,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720980452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:20,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:20,463 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:20,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:20,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:20,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:20,463 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:20,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720980456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,465 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,465 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,465 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,466 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,471 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,471 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,471 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,471 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,471 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:20,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:20,471 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:20,472 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,472 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:20,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,572 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:20,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:20,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:20,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:20,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:20,575 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:20,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:20,576 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:20,576 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,576 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,580 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:20,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:20,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:20,582 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,582 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:20,582 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:20,582 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:20,583 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d442d3a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:20,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:20,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,587 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,588 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:20,588 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:20,588 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:20,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:20,589 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:20,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:20,591 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:20,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:20,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:20,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:20,592 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:20,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,593 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,594 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,596 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:20,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:20,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,597 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:20,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,597 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:20,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,697 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:20,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:20,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:20,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:20,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:20,700 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:20,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:20,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:20,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,703 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:20,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:20,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:20,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:20,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:20,706 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:20,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:20,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15cb6720[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:20,709 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:20,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:20,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:20,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:20,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:20,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:20,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:20,716 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:20,716 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:20,716 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:20,716 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:20,716 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:20,719 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:20,719 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,719 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:20,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,719 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,720 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:20,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:20,722 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:20,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:20,723 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:20,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:20,723 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:20,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:20,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720980757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720980760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720980761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:20,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720980765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,813 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/77825564ca58455aa7bb5c7cfc7bac33 2023-04-05 18:55:20,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/d8dd451ce8de488697d2f2c67c9bad83 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d8dd451ce8de488697d2f2c67c9bad83 2023-04-05 18:55:20,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d8dd451ce8de488697d2f2c67c9bad83, entries=200, sequenceid=240, filesize=32.1 K 2023-04-05 18:55:20,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/26feb215832e4187a706edee2b2cf35b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/26feb215832e4187a706edee2b2cf35b 2023-04-05 18:55:20,823 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:20,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:20,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:20,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:20,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:20,827 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:20,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:20,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:20,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/26feb215832e4187a706edee2b2cf35b, entries=150, sequenceid=240, filesize=11.8 K 2023-04-05 18:55:20,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:20,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:20,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/77825564ca58455aa7bb5c7cfc7bac33 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77825564ca58455aa7bb5c7cfc7bac33 2023-04-05 18:55:20,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:20,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:20,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:20,834 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77825564ca58455aa7bb5c7cfc7bac33, entries=150, sequenceid=240, filesize=11.8 K 2023-04-05 18:55:20,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:20,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:20,834 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:20,834 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:20,834 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:20,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 64f9b0847f90f48ecb0b352322029c8d in 882ms, sequenceid=240, compaction requested=true 2023-04-05 18:55:20,834 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:20,834 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:20,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:20,835 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,835 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:20,835 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:20,835 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:20,835 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:20,835 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:20,836 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-05 18:55:20,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:20,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:20,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:20,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:20,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:20,836 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:20,836 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:20,836 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/A is initiating minor compaction (all files) 2023-04-05 18:55:20,836 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/B is initiating minor compaction (all files) 2023-04-05 18:55:20,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:20,837 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/B in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,836 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/A in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:20,837 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:20,837 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b410e8eb01674ea2b9093c50ace9c805, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ae1fbfa0744a0596b2a4106e04e0bc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/eae144b56cf6435d96d49d21ea0d663f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d8dd451ce8de488697d2f2c67c9bad83] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=108.3 K 2023-04-05 18:55:20,837 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/01939b86d0bd410a9be4a84cc51cbd1e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/639d40d27b5f4e95b5df805b93b5b114, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/14aa804ce87b4a799613ba9187ea9022, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/26feb215832e4187a706edee2b2cf35b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=47.3 K 2023-04-05 18:55:20,837 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting b410e8eb01674ea2b9093c50ace9c805, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680720917204 2023-04-05 18:55:20,837 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 01939b86d0bd410a9be4a84cc51cbd1e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680720917204 2023-04-05 18:55:20,837 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 58ae1fbfa0744a0596b2a4106e04e0bc, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1680720917595 2023-04-05 18:55:20,837 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 639d40d27b5f4e95b5df805b93b5b114, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1680720917595 2023-04-05 18:55:20,838 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting eae144b56cf6435d96d49d21ea0d663f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680720918488 2023-04-05 18:55:20,838 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 14aa804ce87b4a799613ba9187ea9022, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680720918488 2023-04-05 18:55:20,838 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting d8dd451ce8de488697d2f2c67c9bad83, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680720919625 2023-04-05 18:55:20,838 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 26feb215832e4187a706edee2b2cf35b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680720919625 2023-04-05 18:55:20,853 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#B#compaction#345 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:20,861 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#A#compaction#346 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:21,255 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304053b409aadd6164d909ad9d756a4bc3ae7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053b409aadd6164d909ad9d756a4bc3ae7 2023-04-05 18:55:21,260 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=276, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/b1625ceae4ae4cd696e638bb2e54f0ba 2023-04-05 18:55:21,271 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/67d7e5e76b8844edbfc70e460c945cc9 2023-04-05 18:55:21,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:21,273 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:21,276 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/183fac1fcf5e45b9b46af29116b48f3e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/183fac1fcf5e45b9b46af29116b48f3e 2023-04-05 18:55:21,280 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/B of 64f9b0847f90f48ecb0b352322029c8d into 183fac1fcf5e45b9b46af29116b48f3e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:21,280 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:21,280 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/B, priority=12, startTime=1680720920835; duration=0sec 2023-04-05 18:55:21,280 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:21,280 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:21,282 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:21,283 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/C is initiating minor compaction (all files) 2023-04-05 18:55:21,283 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/C in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,283 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c7bf5f7237d540aeb386f866b76e5f97, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/005fcd8da3db4267a80a867902e1d5f8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/ede5786254f146ec8e01dd01a44b7eca, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77825564ca58455aa7bb5c7cfc7bac33] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=47.3 K 2023-04-05 18:55:21,284 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting c7bf5f7237d540aeb386f866b76e5f97, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680720917204 2023-04-05 18:55:21,285 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 005fcd8da3db4267a80a867902e1d5f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1680720917595 2023-04-05 18:55:21,285 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ede5786254f146ec8e01dd01a44b7eca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680720918488 2023-04-05 18:55:21,286 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 77825564ca58455aa7bb5c7cfc7bac33, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680720919625 2023-04-05 18:55:21,288 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/45fc5d86089e4c1597bda41f3d25d307 2023-04-05 18:55:21,296 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/24b3adec06a149a9b8ce14ee442ce358 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/24b3adec06a149a9b8ce14ee442ce358 2023-04-05 18:55:21,297 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/b1625ceae4ae4cd696e638bb2e54f0ba as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b1625ceae4ae4cd696e638bb2e54f0ba 2023-04-05 18:55:21,299 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#C#compaction#349 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:21,303 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b1625ceae4ae4cd696e638bb2e54f0ba, entries=150, sequenceid=276, filesize=25.4 K 2023-04-05 18:55:21,303 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/A of 64f9b0847f90f48ecb0b352322029c8d into 24b3adec06a149a9b8ce14ee442ce358(size=25.7 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:21,303 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:21,303 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/A, priority=12, startTime=1680720920835; duration=0sec 2023-04-05 18:55:21,303 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:21,303 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/67d7e5e76b8844edbfc70e460c945cc9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/67d7e5e76b8844edbfc70e460c945cc9 2023-04-05 18:55:21,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720981298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39952 deadline: 1680720981304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,312 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/67d7e5e76b8844edbfc70e460c945cc9, entries=150, sequenceid=276, filesize=11.9 K 2023-04-05 18:55:21,313 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/45fc5d86089e4c1597bda41f3d25d307 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/45fc5d86089e4c1597bda41f3d25d307 2023-04-05 18:55:21,318 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/45fc5d86089e4c1597bda41f3d25d307, entries=150, sequenceid=276, filesize=11.9 K 2023-04-05 18:55:21,319 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 64f9b0847f90f48ecb0b352322029c8d in 483ms, sequenceid=276, compaction requested=false 2023-04-05 18:55:21,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:21,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:21,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:21,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,319 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/cd9de5a7d99745bc89c3ac40ba703f93 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/cd9de5a7d99745bc89c3ac40ba703f93 2023-04-05 18:55:21,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:21,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,321 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733db4f7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:21,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:21,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,324 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/C of 64f9b0847f90f48ecb0b352322029c8d into cd9de5a7d99745bc89c3ac40ba703f93(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:21,324 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:21,324 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/C, priority=12, startTime=1680720920836; duration=0sec 2023-04-05 18:55:21,324 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:21,326 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,326 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:21,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:21,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:21,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:21,332 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:21,332 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:21,332 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:21,332 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:21,332 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:21,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,336 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,337 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:21,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:21,342 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,343 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,343 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:21,343 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:21,343 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,343 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,343 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:21,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:21,414 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:55:21,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:21,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:21,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:21,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:21,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:21,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:21,443 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:21,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:21,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:21,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:21,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:21,446 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:21,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:21,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:21,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:21,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,448 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:21,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:21,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:21,450 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,450 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:21,450 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:21,450 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:21,450 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:21,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:21,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4039b3b9[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:21,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:21,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,458 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,458 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:21,458 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:21,458 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,458 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:21,458 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:21,459 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:21,462 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:21,462 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:21,462 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:21,462 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:21,462 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:21,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,463 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:21,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,465 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:21,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:21,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,467 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:21,467 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:21,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:21,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720981461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720981468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,567 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:21,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:21,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:21,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:21,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:21,570 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:21,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:21,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:21,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:21,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720981569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,579 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,579 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:21,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:21,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:21,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:21,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:21,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:21,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:21,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c686b79[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:21,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:21,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720981579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,584 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:21,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:21,586 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:21,588 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:21,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:21,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:21,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:21,588 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:21,594 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,594 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,594 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:21,594 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,596 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,598 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:21,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:21,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,599 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:21,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:21,599 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:21,699 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:21,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:21,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:21,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:21,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:21,702 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:21,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:21,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:21,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:21,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:21,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:21,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:21,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:21,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:21,705 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:21,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7254f9b3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:21,710 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:21,710 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,711 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:21,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:21,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:21,716 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:21,716 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:21,716 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:21,716 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:21,716 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:21,718 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,719 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,720 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,723 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:21,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:21,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,723 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:21,723 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:21,724 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,724 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,724 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:21,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:21,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720981761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40034 deadline: 1680720981766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,774 DEBUG [Thread-2343] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:55:21,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720981774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720981775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:21,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720981786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,823 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:21,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:21,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:21,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:21,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:21,827 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:21,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:21,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:21,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:21,832 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,832 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,833 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,833 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,833 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,834 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405488fef021e8a4af0838eaed146f26bcc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405488fef021e8a4af0838eaed146f26bcc 2023-04-05 18:55:21,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:21,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:21,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:21,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:21,836 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:21,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:21,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:21,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:21,837 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,837 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,837 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:21,838 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c507102[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:21,841 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:21,841 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,842 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:21,847 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:21,847 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:21,847 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:21,847 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:21,847 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:21,847 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:21,847 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:21,847 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:21,850 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,850 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,850 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,851 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,852 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,856 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,856 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,857 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:21,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:21,857 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,857 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,857 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,857 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,857 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,857 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,857 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,857 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:21,857 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:21,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:21,957 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:21,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:21,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:21,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:21,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:21,961 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:21,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:21,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:21,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:21,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,964 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:21,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:21,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:21,966 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:21,966 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,967 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:21,967 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:21,967 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:21,967 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:21,968 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,968 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:21,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6179a34a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:21,970 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:21,970 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,971 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:21,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:21,972 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:21,972 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,972 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:21,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,973 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:21,973 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,973 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:21,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:21,975 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:21,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:21,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:21,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:21,975 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:21,978 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:21,978 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,979 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,979 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:21,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,987 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,987 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:21,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:21,987 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:21,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:21,987 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:21,987 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:21,988 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:21,988 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,988 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:21,988 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:21,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:21,988 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:21,988 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:21,988 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:21,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:22,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40028 deadline: 1680720982081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,088 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:22,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:22,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:22,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:22,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:22,091 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:22,091 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,091 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:22,092 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:22,092 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,092 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,093 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,094 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:22,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:22,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720982091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:22,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:22,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:22,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:22,095 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,095 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:22,095 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:22,095 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:22,096 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,096 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,097 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,097 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:22,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e352fbf[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:22,098 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:22,098 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,100 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,100 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:22,100 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:22,100 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:22,100 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,100 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:22,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:22,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:22,101 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:22,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:22,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:22,104 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:22,104 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:22,104 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:22,104 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:22,104 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:22,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:22,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,106 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,107 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,109 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,109 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,109 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,109 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:22,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,109 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:22,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:22,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:22,109 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,110 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,110 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,110 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,110 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,110 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:22,110 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:22,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:22,210 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:22,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:22,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:22,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:22,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:22,213 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:22,213 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,213 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:22,213 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:22,214 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,214 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,216 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:22,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:22,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:22,217 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:22,217 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,217 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:22,217 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:22,218 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:22,218 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,218 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:22,218 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:22,218 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,218 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:22,218 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:22,218 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,218 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:22,218 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:22,219 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,219 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,219 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,219 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:22,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bf9c4ab[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:22,221 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:22,221 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,222 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,222 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:22,222 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:22,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:22,222 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:22,223 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,223 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:22,223 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:22,223 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:22,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:22,225 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:22,225 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:22,225 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:22,225 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:22,225 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:22,230 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,230 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,230 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,231 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,232 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,234 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:22,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:22,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:22,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:22,234 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:22,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,234 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:22,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,235 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,241 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=291, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/58ab193ad21640a3b1e9ae0ee3d37c52 2023-04-05 18:55:22,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/a0ea0262925b43b2974705cb4ba6baf5 2023-04-05 18:55:22,335 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:22,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:22,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:22,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:22,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:22,338 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:22,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:22,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:22,339 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,339 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,340 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,340 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,340 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:22,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:22,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:22,341 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:22,341 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,341 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:22,341 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:22,341 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:22,341 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,341 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:22,341 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:22,341 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:22,344 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,345 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,345 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24ce64b7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:22,346 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:22,346 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,347 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,347 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:22,347 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:22,347 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:22,347 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,347 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:22,348 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,348 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:22,348 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,348 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:22,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:22,350 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:22,350 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:22,350 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:22,350 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:22,350 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:22,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,351 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,352 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:22,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,352 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,353 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,357 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:22,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:22,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:22,357 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,357 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:22,357 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:22,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:22,357 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,357 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:22,357 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,357 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,357 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,357 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,357 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,457 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:22,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:22,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:22,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:22,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:22,461 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:22,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:22,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:22,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,462 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:22,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:22,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:22,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:22,464 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,464 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:22,464 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:22,464 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:22,466 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@881f097[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:22,468 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:22,468 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,469 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,469 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:22,469 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:22,469 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:22,473 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:22,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,473 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:22,473 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:22,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:22,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:22,473 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:22,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:22,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:22,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:22,473 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:22,475 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,475 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,475 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:22,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,477 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:22,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,479 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:22,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:22,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:22,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:22,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,480 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:22,480 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:22,580 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:22,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:22,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:22,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:22,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:22,583 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:22,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:22,583 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:22,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:22,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:22,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:22,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:22,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:22,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:22,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:22,587 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,587 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:22,587 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:22,587 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:22,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:22,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720982583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,589 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,590 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,590 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:22,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ffe6eef[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:22,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:22,592 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,592 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:22,593 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:22,593 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:22,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,593 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:22,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:22,597 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:22,597 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:22,597 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:22,597 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:22,597 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:22,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:22,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:22,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:22,600 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720982595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:22,605 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,605 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,605 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,605 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:22,605 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:22,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:22,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:22,605 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:22,605 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,606 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,606 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:22,606 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:22,606 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,606 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:22,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:22,657 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/0fc175f1c8ec44c5a93716cf955b2c38 2023-04-05 18:55:22,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/58ab193ad21640a3b1e9ae0ee3d37c52 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ab193ad21640a3b1e9ae0ee3d37c52 2023-04-05 18:55:22,665 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ab193ad21640a3b1e9ae0ee3d37c52, entries=200, sequenceid=291, filesize=32.3 K 2023-04-05 18:55:22,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/a0ea0262925b43b2974705cb4ba6baf5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0ea0262925b43b2974705cb4ba6baf5 2023-04-05 18:55:22,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0ea0262925b43b2974705cb4ba6baf5, entries=150, sequenceid=291, filesize=11.9 K 2023-04-05 18:55:22,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/0fc175f1c8ec44c5a93716cf955b2c38 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/0fc175f1c8ec44c5a93716cf955b2c38 2023-04-05 18:55:22,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/0fc175f1c8ec44c5a93716cf955b2c38, entries=150, sequenceid=291, filesize=11.9 K 2023-04-05 18:55:22,673 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 64f9b0847f90f48ecb0b352322029c8d in 1260ms, sequenceid=291, compaction requested=true 2023-04-05 18:55:22,673 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:22,673 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:22,673 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:22,674 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:22,673 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:22,674 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:22,675 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85472 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:22,675 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/A is initiating minor compaction (all files) 2023-04-05 18:55:22,675 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/A in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,675 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/24b3adec06a149a9b8ce14ee442ce358, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b1625ceae4ae4cd696e638bb2e54f0ba, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ab193ad21640a3b1e9ae0ee3d37c52] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=83.5 K 2023-04-05 18:55:22,675 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:22,675 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/B is initiating minor compaction (all files) 2023-04-05 18:55:22,675 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/B in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,676 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/183fac1fcf5e45b9b46af29116b48f3e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/67d7e5e76b8844edbfc70e460c945cc9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0ea0262925b43b2974705cb4ba6baf5] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=36.0 K 2023-04-05 18:55:22,676 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 24b3adec06a149a9b8ce14ee442ce358, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680720919625 2023-04-05 18:55:22,676 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 183fac1fcf5e45b9b46af29116b48f3e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680720919625 2023-04-05 18:55:22,676 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting b1625ceae4ae4cd696e638bb2e54f0ba, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680720920075 2023-04-05 18:55:22,677 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 67d7e5e76b8844edbfc70e460c945cc9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680720920075 2023-04-05 18:55:22,677 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 58ab193ad21640a3b1e9ae0ee3d37c52, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680720921293 2023-04-05 18:55:22,677 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a0ea0262925b43b2974705cb4ba6baf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680720921293 2023-04-05 18:55:22,698 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#B#compaction#354 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:22,702 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#A#compaction#353 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:22,706 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:22,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:22,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:22,712 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:22,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:22,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:22,713 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,713 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:22,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:22,714 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:22,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:22,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:22,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:22,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:22,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:22,725 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:22,725 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:22,726 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:55:22,726 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:22,726 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:22,726 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:22,726 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:22,726 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:22,726 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:23,130 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/036bbb39465f4ef1a66c083d63a00d59 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/036bbb39465f4ef1a66c083d63a00d59 2023-04-05 18:55:23,136 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/B of 64f9b0847f90f48ecb0b352322029c8d into 036bbb39465f4ef1a66c083d63a00d59(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:23,136 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:23,136 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/B, priority=13, startTime=1680720922673; duration=0sec 2023-04-05 18:55:23,136 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:23,136 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:23,138 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:23,138 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/C is initiating minor compaction (all files) 2023-04-05 18:55:23,138 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/C in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:23,138 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/cd9de5a7d99745bc89c3ac40ba703f93, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/45fc5d86089e4c1597bda41f3d25d307, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/0fc175f1c8ec44c5a93716cf955b2c38] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=36.0 K 2023-04-05 18:55:23,139 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting cd9de5a7d99745bc89c3ac40ba703f93, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680720919625 2023-04-05 18:55:23,141 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 45fc5d86089e4c1597bda41f3d25d307, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680720920075 2023-04-05 18:55:23,141 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0fc175f1c8ec44c5a93716cf955b2c38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680720921293 2023-04-05 18:55:23,147 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/d6de259b7bd54cda86d88d537d1dba93 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d6de259b7bd54cda86d88d537d1dba93 2023-04-05 18:55:23,150 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405ff96a5b1ed5b4e6c8e8427713f6c86f7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ff96a5b1ed5b4e6c8e8427713f6c86f7 2023-04-05 18:55:23,152 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#C#compaction#356 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:23,157 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/A of 64f9b0847f90f48ecb0b352322029c8d into d6de259b7bd54cda86d88d537d1dba93(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:23,157 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:23,157 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/A, priority=13, startTime=1680720922673; duration=0sec 2023-04-05 18:55:23,157 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:23,169 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/f6169657d27f439f8c380c7269600d93 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f6169657d27f439f8c380c7269600d93 2023-04-05 18:55:23,182 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/C of 64f9b0847f90f48ecb0b352322029c8d into f6169657d27f439f8c380c7269600d93(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:23,182 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:23,182 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/C, priority=13, startTime=1680720922674; duration=0sec 2023-04-05 18:55:23,182 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:23,562 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/007626940db9483584a75dc7c20817c3 2023-04-05 18:55:23,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:23,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:23,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:23,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720983610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:23,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:23,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720983613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:23,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:23,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720983715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:23,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:23,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720983715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:23,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:23,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720983782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:23,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:23,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720983783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:23,788 DEBUG [Thread-2341] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:55:23,789 DEBUG [Thread-2345] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:55:23,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:23,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720983917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:23,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:23,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720983918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:23,972 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/278f6f3ed1c547fd8f0af7bd7565156c 2023-04-05 18:55:24,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720984222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720984224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,381 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/1bd6998550e943ad8f07b9496c6b35da 2023-04-05 18:55:24,385 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/007626940db9483584a75dc7c20817c3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/007626940db9483584a75dc7c20817c3 2023-04-05 18:55:24,388 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/007626940db9483584a75dc7c20817c3, entries=150, sequenceid=311, filesize=25.4 K 2023-04-05 18:55:24,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/278f6f3ed1c547fd8f0af7bd7565156c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/278f6f3ed1c547fd8f0af7bd7565156c 2023-04-05 18:55:24,392 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/278f6f3ed1c547fd8f0af7bd7565156c, entries=150, sequenceid=311, filesize=11.9 K 2023-04-05 18:55:24,393 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/1bd6998550e943ad8f07b9496c6b35da as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1bd6998550e943ad8f07b9496c6b35da 2023-04-05 18:55:24,396 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1bd6998550e943ad8f07b9496c6b35da, entries=150, sequenceid=311, filesize=11.9 K 2023-04-05 18:55:24,397 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 64f9b0847f90f48ecb0b352322029c8d in 1671ms, sequenceid=311, compaction requested=false 2023-04-05 18:55:24,397 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:24,397 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:24,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:24,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:24,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:24,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:24,399 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,399 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a272927[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:24,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:24,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:24,402 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:24,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:24,403 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:24,403 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:24,403 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,403 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:24,404 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:24,406 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:24,406 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:24,406 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:24,406 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:24,406 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:24,408 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:24,408 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:24,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:24,408 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:24,408 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:24,409 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:24,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:24,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:24,413 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:24,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:24,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:24,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:24,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:24,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:24,413 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:24,414 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:24,414 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:24,414 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:24,414 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:24,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:24,514 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:24,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:24,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:24,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:24,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:24,517 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:24,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:24,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:24,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:24,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:24,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:24,518 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,518 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:24,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:24,519 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,519 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:24,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:24,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:24,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:24,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:24,520 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:24,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:24,728 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:24,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:24,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720984775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:24,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720984781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:24,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720984882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:24,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720984886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:24,933 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405659d17eaf0ef4990a86e2ad6d6d0fd09 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405659d17eaf0ef4990a86e2ad6d6d0fd09 2023-04-05 18:55:25,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:25,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720985087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:25,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:25,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720985090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:25,338 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=326, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/05dd2d2a9fdc44f28500bda4eb037319 2023-04-05 18:55:25,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:25,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720985393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:25,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:25,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720985394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:25,752 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/385892aa01ad4f78a683c58e635aaced 2023-04-05 18:55:25,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:25,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40034 deadline: 1680720985800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:25,807 DEBUG [Thread-2343] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8198 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., hostname=jenkins-hbase4.apache.org,33489,1680720816595, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 18:55:25,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:25,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720985901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:25,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:25,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720985903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,162 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/a533cf366ca74fb6b5a3673e4dbab725 2023-04-05 18:55:26,166 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/05dd2d2a9fdc44f28500bda4eb037319 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/05dd2d2a9fdc44f28500bda4eb037319 2023-04-05 18:55:26,169 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/05dd2d2a9fdc44f28500bda4eb037319, entries=150, sequenceid=326, filesize=25.4 K 2023-04-05 18:55:26,170 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/385892aa01ad4f78a683c58e635aaced as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/385892aa01ad4f78a683c58e635aaced 2023-04-05 18:55:26,173 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/385892aa01ad4f78a683c58e635aaced, entries=150, sequenceid=326, filesize=11.9 K 2023-04-05 18:55:26,174 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/a533cf366ca74fb6b5a3673e4dbab725 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/a533cf366ca74fb6b5a3673e4dbab725 2023-04-05 18:55:26,177 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/a533cf366ca74fb6b5a3673e4dbab725, entries=150, sequenceid=326, filesize=11.9 K 2023-04-05 18:55:26,177 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 64f9b0847f90f48ecb0b352322029c8d in 1657ms, sequenceid=326, compaction requested=true 2023-04-05 18:55:26,177 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:26,177 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:26,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:26,179 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:26,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:26,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:26,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:26,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d9b1908[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:26,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:26,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:26,183 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:26,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:26,183 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:26,183 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:26,183 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,183 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:26,183 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:26,184 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:26,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:26,186 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:26,186 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:26,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:26,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:26,187 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:26,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:26,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:26,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:26,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:26,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:26,190 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:26,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,192 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,192 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:26,192 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:26,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:26,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:26,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:26,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:26,192 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:26,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:26,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:26,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,193 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:26,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:26,193 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:26,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:26,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:26,293 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:26,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:26,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:26,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:26,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:26,297 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:26,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:26,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:26,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:26,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:26,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:26,298 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,298 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:26,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:26,299 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,299 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:26,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:26,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:26,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:26,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:26,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:26,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:26,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:26,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:26,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:26,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:26,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:26,300 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:55:26,301 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:26,301 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:26,301 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:26,301 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:26,301 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:26,301 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:26,317 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405bfcb689b53f24928a9aa7f1222b56aae to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bfcb689b53f24928a9aa7f1222b56aae 2023-04-05 18:55:26,723 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=346, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/14d95cd0b8f24aed93c75520a46745fc 2023-04-05 18:55:26,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:26,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:26,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:26,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720986928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:26,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:26,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720986931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:27,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720987032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:27,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720987038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,133 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/9b2a4e5d0c604dd08dada9874f972d35 2023-04-05 18:55:27,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:27,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720987239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:27,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720987245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,543 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/959e779706a241dcb395fa0f2d6f34e6 2023-04-05 18:55:27,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:27,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720987542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,547 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/14d95cd0b8f24aed93c75520a46745fc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/14d95cd0b8f24aed93c75520a46745fc 2023-04-05 18:55:27,550 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/14d95cd0b8f24aed93c75520a46745fc, entries=150, sequenceid=346, filesize=25.4 K 2023-04-05 18:55:27,551 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/9b2a4e5d0c604dd08dada9874f972d35 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/9b2a4e5d0c604dd08dada9874f972d35 2023-04-05 18:55:27,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:27,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720987549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,554 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/9b2a4e5d0c604dd08dada9874f972d35, entries=150, sequenceid=346, filesize=11.9 K 2023-04-05 18:55:27,555 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/959e779706a241dcb395fa0f2d6f34e6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/959e779706a241dcb395fa0f2d6f34e6 2023-04-05 18:55:27,558 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/959e779706a241dcb395fa0f2d6f34e6, entries=150, sequenceid=346, filesize=11.9 K 2023-04-05 18:55:27,558 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 64f9b0847f90f48ecb0b352322029c8d in 1258ms, sequenceid=346, compaction requested=true 2023-04-05 18:55:27,559 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:27,559 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:27,561 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,561 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a995964[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:27,563 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:27,563 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:27,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:27,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:27,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:27,566 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:27,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:27,568 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:27,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:27,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:27,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:27,568 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:27,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:27,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:27,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:27,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:27,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:27,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:27,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:27,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:27,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:27,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:27,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:27,577 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:27,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:27,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:27,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:27,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:27,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:27,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:27,578 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:27,578 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:27,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:27,678 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:27,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:27,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:27,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:27,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:27,681 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:27,681 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:27,681 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:27,681 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:27,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:27,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:27,683 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:27,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:27,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:27,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:27,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:27,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:27,684 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:27,684 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:27,685 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:55:27,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:27,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:27,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:27,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:27,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:27,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:27,696 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405470887fc699e4f26ba6b97c27d49a33e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405470887fc699e4f26ba6b97c27d49a33e 2023-04-05 18:55:27,703 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/0924e8ee688944b59421a3a1a773c013 2023-04-05 18:55:27,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:27,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:27,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:27,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720987866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:27,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720987877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:27,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720987978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:27,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:27,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720987981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:28,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720988050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:28,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720988055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,122 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/794acd836fa942c9822258a6d1aea046 2023-04-05 18:55:28,149 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/b94b1fd549d44c109f75e854b41a0a1d 2023-04-05 18:55:28,154 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/0924e8ee688944b59421a3a1a773c013 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/0924e8ee688944b59421a3a1a773c013 2023-04-05 18:55:28,158 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/0924e8ee688944b59421a3a1a773c013, entries=150, sequenceid=358, filesize=25.4 K 2023-04-05 18:55:28,158 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/794acd836fa942c9822258a6d1aea046 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/794acd836fa942c9822258a6d1aea046 2023-04-05 18:55:28,163 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/794acd836fa942c9822258a6d1aea046, entries=150, sequenceid=358, filesize=11.9 K 2023-04-05 18:55:28,164 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/b94b1fd549d44c109f75e854b41a0a1d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/b94b1fd549d44c109f75e854b41a0a1d 2023-04-05 18:55:28,182 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/b94b1fd549d44c109f75e854b41a0a1d, entries=150, sequenceid=358, filesize=11.9 K 2023-04-05 18:55:28,182 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 64f9b0847f90f48ecb0b352322029c8d in 497ms, sequenceid=358, compaction requested=true 2023-04-05 18:55:28,182 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:28,182 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:28,183 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:28,183 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,183 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:28,183 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:28,185 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:28,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26dccbce[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:28,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:28,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:28,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,189 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:28,189 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:28,189 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:28,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:28,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-05 18:55:28,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:28,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:28,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:28,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:28,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:28,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:28,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:28,200 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:28,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:28,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:28,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:28,200 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:28,202 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,202 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:28,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,202 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,202 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,204 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:28,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:28,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:28,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:28,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:28,206 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:28,206 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:28,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:28,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:28,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:28,214 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304055833e9a7a1ca4257a5134e7858b7d0e9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055833e9a7a1ca4257a5134e7858b7d0e9 2023-04-05 18:55:28,221 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=379, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/2a03edb7b1bf41e594dad887e2a35fd0 2023-04-05 18:55:28,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:28,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720988230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:28,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720988231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,306 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:28,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:28,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:28,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:28,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:28,310 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:28,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:28,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:28,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:28,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,312 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:28,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:28,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:28,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:28,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:28,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:28,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:28,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:28,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:28,313 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:28,316 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,316 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25b1a023[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:28,317 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:28,317 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:28,319 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:28,321 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:28,324 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:28,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:28,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:28,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:28,325 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:28,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:28,326 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:28,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:28,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:28,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:28,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,332 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:28,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:28,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:28,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,332 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:28,333 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:28,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:28,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720988334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:28,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720988339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,433 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:28,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:28,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:28,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:28,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:28,438 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:28,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:28,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:28,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:28,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,440 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:28,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:28,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:28,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:28,442 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:28,442 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:28,442 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:28,442 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:28,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:28,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:28,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:28,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:28,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:28,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1057bec4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:28,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:28,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,446 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,446 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:28,446 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:28,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:28,446 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:28,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:28,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:28,448 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,448 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:28,450 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:28,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:28,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:28,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:28,451 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:28,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:28,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,454 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:28,455 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:28,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:28,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:28,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:28,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:28,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:28,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,456 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,456 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,456 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,456 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,456 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:28,456 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:28,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:28,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:28,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1680720988540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:28,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720988548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,556 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:28,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:28,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:28,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:28,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:28,560 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:28,560 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:28,560 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:28,560 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:28,560 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,560 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,561 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,561 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:28,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:28,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:28,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:28,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:28,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:28,563 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:28,563 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:28,563 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:28,563 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:28,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,569 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,569 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,569 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:28,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:28,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a597d3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:28,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,572 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:28,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:28,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:28,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:28,573 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,573 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:28,573 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,573 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:28,575 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:28,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:28,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:28,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:28,575 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:28,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:28,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:28,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:28,577 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:28,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:28,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:28,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:28,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,582 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:28,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:28,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:28,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:28,582 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:28,583 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:28,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/2908abc1651048d7a6f2d63b2f232d3e 2023-04-05 18:55:28,649 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/fe5998047235427baaa04938197f9c32 2023-04-05 18:55:28,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/2a03edb7b1bf41e594dad887e2a35fd0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/2a03edb7b1bf41e594dad887e2a35fd0 2023-04-05 18:55:28,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/2a03edb7b1bf41e594dad887e2a35fd0, entries=200, sequenceid=379, filesize=32.3 K 2023-04-05 18:55:28,657 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/2908abc1651048d7a6f2d63b2f232d3e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/2908abc1651048d7a6f2d63b2f232d3e 2023-04-05 18:55:28,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/2908abc1651048d7a6f2d63b2f232d3e, entries=150, sequenceid=379, filesize=11.9 K 2023-04-05 18:55:28,663 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/fe5998047235427baaa04938197f9c32 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/fe5998047235427baaa04938197f9c32 2023-04-05 18:55:28,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/fe5998047235427baaa04938197f9c32, entries=150, sequenceid=379, filesize=11.9 K 2023-04-05 18:55:28,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 64f9b0847f90f48ecb0b352322029c8d in 475ms, sequenceid=379, compaction requested=true 2023-04-05 18:55:28,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:28,667 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:28,667 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 18:55:28,667 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:28,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:55:28,668 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 18:55:28,669 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163862 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 18:55:28,669 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/A is initiating minor compaction (all files) 2023-04-05 18:55:28,669 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/A in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:28,669 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d6de259b7bd54cda86d88d537d1dba93, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/007626940db9483584a75dc7c20817c3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/05dd2d2a9fdc44f28500bda4eb037319, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/14d95cd0b8f24aed93c75520a46745fc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/0924e8ee688944b59421a3a1a773c013, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/2a03edb7b1bf41e594dad887e2a35fd0] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=160.0 K 2023-04-05 18:55:28,670 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting d6de259b7bd54cda86d88d537d1dba93, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680720921293 2023-04-05 18:55:28,670 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 007626940db9483584a75dc7c20817c3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680720921454 2023-04-05 18:55:28,670 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 18:55:28,670 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/B is initiating minor compaction (all files) 2023-04-05 18:55:28,670 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/B in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:28,671 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/036bbb39465f4ef1a66c083d63a00d59, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/278f6f3ed1c547fd8f0af7bd7565156c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/385892aa01ad4f78a683c58e635aaced, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/9b2a4e5d0c604dd08dada9874f972d35, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/794acd836fa942c9822258a6d1aea046, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/2908abc1651048d7a6f2d63b2f232d3e] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=71.9 K 2023-04-05 18:55:28,671 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 036bbb39465f4ef1a66c083d63a00d59, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680720921293 2023-04-05 18:55:28,671 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 05dd2d2a9fdc44f28500bda4eb037319, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680720923600 2023-04-05 18:55:28,672 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 278f6f3ed1c547fd8f0af7bd7565156c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680720921454 2023-04-05 18:55:28,672 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 14d95cd0b8f24aed93c75520a46745fc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680720924767 2023-04-05 18:55:28,672 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 385892aa01ad4f78a683c58e635aaced, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680720923600 2023-04-05 18:55:28,673 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 0924e8ee688944b59421a3a1a773c013, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680720926920 2023-04-05 18:55:28,673 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 9b2a4e5d0c604dd08dada9874f972d35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680720924767 2023-04-05 18:55:28,673 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 2a03edb7b1bf41e594dad887e2a35fd0, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680720927830 2023-04-05 18:55:28,673 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 794acd836fa942c9822258a6d1aea046, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680720926920 2023-04-05 18:55:28,674 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 2908abc1651048d7a6f2d63b2f232d3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680720927830 2023-04-05 18:55:28,683 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:28,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:28,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:28,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:28,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:28,692 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:28,692 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:28,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:28,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:28,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,695 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:28,695 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:28,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:28,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:28,699 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#B#compaction#372 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:28,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:28,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:28,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:28,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:28,709 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:28,709 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:28,709 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:55:28,709 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:28,709 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:28,709 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:28,709 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:28,709 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:28,709 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:28,719 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#A#compaction#371 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:28,754 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304055491c2d0fab2475b802f90c0775b89a2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055491c2d0fab2475b802f90c0775b89a2 2023-04-05 18:55:28,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:28,854 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:28,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:28,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40018 deadline: 1680720988929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:28,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720928889, max timestamp=1680720928941Num uniques 859; 2023-04-05 18:55:28,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720928889, max timestamp=1680720928941Num uniques 859; 2023-04-05 18:55:28,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720928889, max timestamp=1680720928941Num uniques 859; 2023-04-05 18:55:28,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:28,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:28,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:28,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1680720988942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1680720989086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40018 deadline: 1680720989089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720989098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720989099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,129 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/48830f20fc884764a26f0a92e7e0f45c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/48830f20fc884764a26f0a92e7e0f45c 2023-04-05 18:55:29,133 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/B of 64f9b0847f90f48ecb0b352322029c8d into 48830f20fc884764a26f0a92e7e0f45c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:29,133 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:29,133 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/B, priority=10, startTime=1680720928667; duration=0sec 2023-04-05 18:55:29,133 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:29,133 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 18:55:29,135 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 18:55:29,135 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/C is initiating minor compaction (all files) 2023-04-05 18:55:29,135 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/C in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:29,135 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f6169657d27f439f8c380c7269600d93, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1bd6998550e943ad8f07b9496c6b35da, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/a533cf366ca74fb6b5a3673e4dbab725, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/959e779706a241dcb395fa0f2d6f34e6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/b94b1fd549d44c109f75e854b41a0a1d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/fe5998047235427baaa04938197f9c32] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=71.9 K 2023-04-05 18:55:29,135 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting f6169657d27f439f8c380c7269600d93, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680720921293 2023-04-05 18:55:29,136 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1bd6998550e943ad8f07b9496c6b35da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680720921454 2023-04-05 18:55:29,136 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a533cf366ca74fb6b5a3673e4dbab725, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680720923600 2023-04-05 18:55:29,136 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 959e779706a241dcb395fa0f2d6f34e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680720924767 2023-04-05 18:55:29,137 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting b94b1fd549d44c109f75e854b41a0a1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680720926920 2023-04-05 18:55:29,137 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting fe5998047235427baaa04938197f9c32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680720927830 2023-04-05 18:55:29,159 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#C#compaction#374 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:29,161 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=390, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/5a8e823913714de1a7b4176f118484e0 2023-04-05 18:55:29,171 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/7fcc4441f5ff4b468cb6f919b77adb88 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/7fcc4441f5ff4b468cb6f919b77adb88 2023-04-05 18:55:29,179 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/C of 64f9b0847f90f48ecb0b352322029c8d into 7fcc4441f5ff4b468cb6f919b77adb88(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:29,180 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:29,180 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/C, priority=10, startTime=1680720928668; duration=0sec 2023-04-05 18:55:29,180 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:29,191 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/177c1e55bae841d0b54c3372e4976d0e 2023-04-05 18:55:29,194 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/742389e3bb544122b7bc5d4b00626e58 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/742389e3bb544122b7bc5d4b00626e58 2023-04-05 18:55:29,199 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/A of 64f9b0847f90f48ecb0b352322029c8d into 742389e3bb544122b7bc5d4b00626e58(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:29,199 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:29,199 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/A, priority=10, startTime=1680720928667; duration=0sec 2023-04-05 18:55:29,199 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:29,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1680720989200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40018 deadline: 1680720989201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720989210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720989210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1680720989405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40018 deadline: 1680720989411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720989411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720989425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,604 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/58fb790acd624960b5f9e269deec1f5e 2023-04-05 18:55:29,608 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/5a8e823913714de1a7b4176f118484e0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/5a8e823913714de1a7b4176f118484e0 2023-04-05 18:55:29,611 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/5a8e823913714de1a7b4176f118484e0, entries=150, sequenceid=390, filesize=25.4 K 2023-04-05 18:55:29,611 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/177c1e55bae841d0b54c3372e4976d0e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/177c1e55bae841d0b54c3372e4976d0e 2023-04-05 18:55:29,615 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/177c1e55bae841d0b54c3372e4976d0e, entries=150, sequenceid=390, filesize=11.9 K 2023-04-05 18:55:29,615 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/58fb790acd624960b5f9e269deec1f5e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/58fb790acd624960b5f9e269deec1f5e 2023-04-05 18:55:29,618 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/58fb790acd624960b5f9e269deec1f5e, entries=150, sequenceid=390, filesize=11.9 K 2023-04-05 18:55:29,619 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 64f9b0847f90f48ecb0b352322029c8d in 910ms, sequenceid=390, compaction requested=false 2023-04-05 18:55:29,619 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:29,619 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:29,619 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:29,619 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:29,619 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:29,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:29,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:29,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:29,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7067b8b2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:29,630 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:29,630 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,631 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:29,632 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,632 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:29,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,634 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:29,634 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:29,634 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:29,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:29,638 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:29,638 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:29,638 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:29,638 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:29,638 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:29,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:29,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:29,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:29,640 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,641 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:29,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:29,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:29,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:29,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:29,644 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:29,644 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:29,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,644 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:29,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:29,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-05 18:55:29,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:29,713 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:29,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:29,713 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:29,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:29,713 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:29,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40018 deadline: 1680720989726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720989729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720989730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1680720989731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,744 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:29,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:29,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:29,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:29,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:29,755 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:29,755 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:29,755 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:29,755 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:29,755 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,756 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,757 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,757 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:29,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:29,757 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,757 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:29,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:29,762 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:29,762 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:29,762 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:29,762 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:29,763 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:29,763 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,763 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:29,763 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:29,766 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:29,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:29,768 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:29,768 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58ffae25[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:29,769 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,769 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:29,769 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:29,769 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,769 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:29,769 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:29,770 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:29,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:29,772 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:29,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:29,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:29,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:29,773 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:29,774 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,774 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:29,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,774 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:29,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:29,775 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,779 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,779 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,779 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,779 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,779 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,779 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,779 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,779 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:29,779 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:29,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:29,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:29,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:29,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:29,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:29,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:29,780 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:29,781 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:29,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40018 deadline: 1680720989833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720989834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:29,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1680720989838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,881 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:29,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:29,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:29,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:29,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:29,884 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:29,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:29,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:29,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:29,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,886 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,886 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:29,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:29,886 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,887 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:29,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:29,888 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:29,888 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:29,888 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:29,888 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,889 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:29,890 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,890 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:29,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75f38a60[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:29,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,893 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:29,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:29,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:29,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,894 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:29,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:29,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:29,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:29,899 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:29,901 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:29,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:29,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:29,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:29,901 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:29,902 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,902 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,902 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:29,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,903 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:29,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,904 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:29,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:29,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:29,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:29,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:29,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:29,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:29,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:29,907 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:29,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:29,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:29,908 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:29,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:29,908 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:30,008 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:30,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:30,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:30,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:30,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:30,011 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:30,011 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:30,011 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:30,011 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:30,012 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,012 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:30,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:30,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:30,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:30,014 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:30,014 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:30,014 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:30,015 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:30,015 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:30,015 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:30,015 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,015 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:30,015 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:30,016 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69749002[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:30,018 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:30,018 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,020 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:30,020 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,020 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:30,021 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,021 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:30,021 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,021 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:30,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:30,023 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:30,023 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:30,023 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:30,023 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:30,023 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:30,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,028 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,029 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,029 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:30,029 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,030 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:30,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:30,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:30,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:30,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:30,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,030 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:30,030 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:30,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:30,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,030 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:30,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40018 deadline: 1680720990038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:30,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720990038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:30,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1680720990044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,125 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040579f291949ae943f7a08c7793f77ee33f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040579f291949ae943f7a08c7793f77ee33f 2023-04-05 18:55:30,130 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:30,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:30,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:30,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:30,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:30,134 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:30,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:30,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:30,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:30,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,136 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,136 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:30,136 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,136 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:30,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:30,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:30,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:30,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:30,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:30,137 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:30,137 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:30,137 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:30,137 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:30,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:30,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:30,137 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,138 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:30,138 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:30,139 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,139 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@241b07b6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:30,141 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:30,141 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,142 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,142 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:30,142 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,143 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:30,142 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:30,143 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:30,144 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,144 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:30,144 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,144 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:30,146 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:30,146 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:30,146 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:30,146 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:30,146 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:30,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:30,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,148 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,149 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,151 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:30,151 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:30,151 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:30,151 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,152 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,152 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,152 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,152 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:30,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:30,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:30,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:30,152 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,153 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:30,153 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:30,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:30,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:30,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720990237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,253 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:30,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:30,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:30,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:30,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:30,256 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:30,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:30,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:30,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:30,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:30,258 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:30,258 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,258 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:30,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:30,259 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:30,259 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:30,259 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:30,259 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:30,259 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:30,261 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:30,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b00b96[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:30,263 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:30,263 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,265 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,265 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:30,265 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:30,265 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:30,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:30,266 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,266 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,266 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:30,267 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:30,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:30,269 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:30,269 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:30,269 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:30,269 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:30,269 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:30,273 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,273 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,273 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:30,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:30,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:30,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:30,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:30,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:30,279 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:30,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:30,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,279 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:30,279 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:30,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:30,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:30,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40018 deadline: 1680720990343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:30,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720990344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:30,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1680720990348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,379 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:30,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:30,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:30,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:30,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:30,382 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:30,382 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:30,382 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:30,382 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:30,383 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,383 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,384 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,384 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:30,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:30,384 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,384 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:30,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:30,385 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:30,385 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:30,385 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:30,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:30,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:30,386 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:30,386 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:30,386 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:30,386 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:30,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:30,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:30,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:30,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,387 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f9214a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:30,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:30,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,390 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:30,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:30,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:30,390 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,390 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:30,392 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:30,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:30,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:30,395 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:30,395 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:30,395 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:30,395 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:30,395 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:30,396 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,396 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,396 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,397 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,399 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,401 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,401 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,401 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:30,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:30,401 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,401 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:30,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:30,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:30,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:30,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:30,402 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,402 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,402 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:30,402 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,402 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:30,402 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:30,502 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:30,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:30,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:30,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:30,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:30,505 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:30,505 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:30,505 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:30,505 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:30,506 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,506 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,507 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,507 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:30,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:30,507 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,507 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:30,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:30,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:30,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:30,509 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:30,509 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. started... 2023-04-05 18:55:30,509 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. as already flushing 2023-04-05 18:55:30,509 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:30,511 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,511 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:30,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254048d7[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:30,513 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:30,513 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,514 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,514 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:30,514 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:30,514 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:30,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:30,515 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,515 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:30,515 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,515 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:30,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:30,517 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:30,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:30,518 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:30,518 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:30,518 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:30,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:30,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:30,519 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,520 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,524 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,524 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,524 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:30,524 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:30,524 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:30,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:30,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:30,524 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:30,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:30,524 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:30,524 DEBUG [Thread-2351] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:30,524 DEBUG [Thread-2351] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:30,524 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,524 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:30,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:30,524 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:30,530 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=430, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/7357e976065e48c99b8517ad178e1688 2023-04-05 18:55:30,541 DEBUG [Thread-2358] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eb8f142 to 127.0.0.1:49755 2023-04-05 18:55:30,541 DEBUG [Thread-2358] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:30,543 DEBUG [Thread-2356] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:49755 2023-04-05 18:55:30,543 DEBUG [Thread-2354] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:49755 2023-04-05 18:55:30,543 DEBUG [Thread-2356] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:30,543 DEBUG [Thread-2354] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:30,549 DEBUG [Thread-2360] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ce79633 to 127.0.0.1:49755 2023-04-05 18:55:30,549 DEBUG [Thread-2360] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:30,549 DEBUG [Thread-2352] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:49755 2023-04-05 18:55:30,549 DEBUG [Thread-2352] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:30,624 DEBUG [Thread-2351] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:30,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:30,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:30,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39952 deadline: 1680720990849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:30,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40018 deadline: 1680720990850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:30,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1680720990860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:30,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/773b7255a01b415d850667246a7a3cd7 2023-04-05 18:55:31,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:31,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40028 deadline: 1680720991248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:31,350 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/8c32d5ea677e4d9ebfc0451a9e44e4f1 2023-04-05 18:55:31,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/7357e976065e48c99b8517ad178e1688 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/7357e976065e48c99b8517ad178e1688 2023-04-05 18:55:31,358 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/7357e976065e48c99b8517ad178e1688, entries=200, sequenceid=430, filesize=32.3 K 2023-04-05 18:55:31,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/773b7255a01b415d850667246a7a3cd7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/773b7255a01b415d850667246a7a3cd7 2023-04-05 18:55:31,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/773b7255a01b415d850667246a7a3cd7, entries=150, sequenceid=430, filesize=11.9 K 2023-04-05 18:55:31,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/8c32d5ea677e4d9ebfc0451a9e44e4f1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/8c32d5ea677e4d9ebfc0451a9e44e4f1 2023-04-05 18:55:31,366 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/8c32d5ea677e4d9ebfc0451a9e44e4f1, entries=150, sequenceid=430, filesize=11.9 K 2023-04-05 18:55:31,366 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 64f9b0847f90f48ecb0b352322029c8d in 1653ms, sequenceid=430, compaction requested=true 2023-04-05 18:55:31,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:31,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:31,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:31,367 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:31,367 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:31,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:31,368 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85928 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:31,368 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/A is initiating minor compaction (all files) 2023-04-05 18:55:31,368 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:31,368 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/A in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:31,368 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/B is initiating minor compaction (all files) 2023-04-05 18:55:31,368 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/742389e3bb544122b7bc5d4b00626e58, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/5a8e823913714de1a7b4176f118484e0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/7357e976065e48c99b8517ad178e1688] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=83.9 K 2023-04-05 18:55:31,368 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/B in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:31,368 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/48830f20fc884764a26f0a92e7e0f45c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/177c1e55bae841d0b54c3372e4976d0e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/773b7255a01b415d850667246a7a3cd7] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=36.4 K 2023-04-05 18:55:31,368 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 742389e3bb544122b7bc5d4b00626e58, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680720927830 2023-04-05 18:55:31,368 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 48830f20fc884764a26f0a92e7e0f45c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680720927830 2023-04-05 18:55:31,368 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 5a8e823913714de1a7b4176f118484e0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680720928213 2023-04-05 18:55:31,368 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 177c1e55bae841d0b54c3372e4976d0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680720928213 2023-04-05 18:55:31,368 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7357e976065e48c99b8517ad178e1688, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680720929088 2023-04-05 18:55:31,369 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 773b7255a01b415d850667246a7a3cd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680720929088 2023-04-05 18:55:31,377 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#B#compaction#381 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:31,378 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#A#compaction#380 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:31,798 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/2edaa0cee5914d0ca904e784bcae1ef5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/2edaa0cee5914d0ca904e784bcae1ef5 2023-04-05 18:55:31,802 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/B of 64f9b0847f90f48ecb0b352322029c8d into 2edaa0cee5914d0ca904e784bcae1ef5(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:31,802 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:31,802 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/B, priority=13, startTime=1680720931367; duration=0sec 2023-04-05 18:55:31,802 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:31,802 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:31,802 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:31,803 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 64f9b0847f90f48ecb0b352322029c8d/C is initiating minor compaction (all files) 2023-04-05 18:55:31,803 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64f9b0847f90f48ecb0b352322029c8d/C in TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:31,803 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/7fcc4441f5ff4b468cb6f919b77adb88, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/58fb790acd624960b5f9e269deec1f5e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/8c32d5ea677e4d9ebfc0451a9e44e4f1] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp, totalSize=36.4 K 2023-04-05 18:55:31,803 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 7fcc4441f5ff4b468cb6f919b77adb88, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680720927830 2023-04-05 18:55:31,803 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 58fb790acd624960b5f9e269deec1f5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680720928213 2023-04-05 18:55:31,803 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 8c32d5ea677e4d9ebfc0451a9e44e4f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680720929088 2023-04-05 18:55:31,808 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 64f9b0847f90f48ecb0b352322029c8d#C#compaction#382 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:31,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:31,860 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:55:31,860 DEBUG [Thread-2345] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:49755 2023-04-05 18:55:31,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:31,860 DEBUG [Thread-2345] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:31,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:31,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:31,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:31,860 DEBUG [Thread-2349] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:49755 2023-04-05 18:55:31,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:31,860 DEBUG [Thread-2349] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:31,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:31,871 DEBUG [Thread-2341] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:49755 2023-04-05 18:55:31,871 DEBUG [Thread-2341] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:32,214 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/be2378e164ec4db5acb1a92931a1b150 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/be2378e164ec4db5acb1a92931a1b150 2023-04-05 18:55:32,218 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/C of 64f9b0847f90f48ecb0b352322029c8d into be2378e164ec4db5acb1a92931a1b150(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:32,218 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:32,218 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/C, priority=13, startTime=1680720931367; duration=0sec 2023-04-05 18:55:32,218 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:32,269 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304053c0cfca071c3430a8d26dd1a7447d35d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053c0cfca071c3430a8d26dd1a7447d35d 2023-04-05 18:55:32,607 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/43060306d31f442db1177bd100f8ad1a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/43060306d31f442db1177bd100f8ad1a 2023-04-05 18:55:32,610 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64f9b0847f90f48ecb0b352322029c8d/A of 64f9b0847f90f48ecb0b352322029c8d into 43060306d31f442db1177bd100f8ad1a(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 18:55:32,610 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:32,610 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d., storeName=64f9b0847f90f48ecb0b352322029c8d/A, priority=13, startTime=1680720931367; duration=1sec 2023-04-05 18:55:32,610 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:32,674 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=443, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/7eabf80323164bb284a7976a7f764eac 2023-04-05 18:55:33,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/c5226bdd4e9647adae6e7cf378b64e92 2023-04-05 18:55:33,260 DEBUG [Thread-2347] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06a0df34 to 127.0.0.1:49755 2023-04-05 18:55:33,260 DEBUG [Thread-2347] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:33,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/9fd79770be8944349393115ac73ccb3c 2023-04-05 18:55:33,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/7eabf80323164bb284a7976a7f764eac as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/7eabf80323164bb284a7976a7f764eac 2023-04-05 18:55:33,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/7eabf80323164bb284a7976a7f764eac, entries=150, sequenceid=443, filesize=25.4 K 2023-04-05 18:55:33,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/c5226bdd4e9647adae6e7cf378b64e92 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/c5226bdd4e9647adae6e7cf378b64e92 2023-04-05 18:55:33,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/c5226bdd4e9647adae6e7cf378b64e92, entries=150, sequenceid=443, filesize=11.9 K 2023-04-05 18:55:33,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/9fd79770be8944349393115ac73ccb3c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/9fd79770be8944349393115ac73ccb3c 2023-04-05 18:55:33,502 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/9fd79770be8944349393115ac73ccb3c, entries=150, sequenceid=443, filesize=11.9 K 2023-04-05 18:55:33,503 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=13.42 KB/13740 for 64f9b0847f90f48ecb0b352322029c8d in 1643ms, sequenceid=443, compaction requested=false 2023-04-05 18:55:33,503 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:35,850 DEBUG [Thread-2343] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:49755 2023-04-05 18:55:35,851 DEBUG [Thread-2343] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 90 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1808 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5424 rows 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1803 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5409 rows 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1805 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5412 rows 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1794 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5381 rows 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1795 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5380 rows 2023-04-05 18:55:35,851 INFO [Listener at localhost/43705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 18:55:35,852 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:49755 2023-04-05 18:55:35,852 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:35,855 INFO [Listener at localhost/43705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 18:55:35,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 18:55:35,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:35,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 18:55:35,859 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720935859"}]},"ts":"1680720935859"} 2023-04-05 18:55:35,860 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 18:55:35,861 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 18:55:35,862 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, UNASSIGN}] 2023-04-05 18:55:35,863 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, UNASSIGN 2023-04-05 18:55:35,863 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=64f9b0847f90f48ecb0b352322029c8d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:35,864 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720935863"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720935863"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720935863"}]},"ts":"1680720935863"} 2023-04-05 18:55:35,864 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:55:35,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 18:55:36,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:36,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 64f9b0847f90f48ecb0b352322029c8d, disabling compactions & flushes 2023-04-05 18:55:36,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:36,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:36,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. after waiting 0 ms 2023-04-05 18:55:36,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:36,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 64f9b0847f90f48ecb0b352322029c8d 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-05 18:55:36,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=A 2023-04-05 18:55:36,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:36,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=B 2023-04-05 18:55:36,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:36,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64f9b0847f90f48ecb0b352322029c8d, store=C 2023-04-05 18:55:36,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:36,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 18:55:36,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304053c482cc05b684bca84f143e7c88b44c6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053c482cc05b684bca84f143e7c88b44c6 2023-04-05 18:55:36,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 18:55:36,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=451, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/b2e74cde43e64bf393483fd7cd828166 2023-04-05 18:55:36,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/e39d6e02d9b849aeb92e33be5fac44cc 2023-04-05 18:55:36,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 18:55:37,245 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/c2140b914c7c4f38bf6f7526d78ef18b 2023-04-05 18:55:37,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/A/b2e74cde43e64bf393483fd7cd828166 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b2e74cde43e64bf393483fd7cd828166 2023-04-05 18:55:37,251 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b2e74cde43e64bf393483fd7cd828166, entries=50, sequenceid=451, filesize=11.7 K 2023-04-05 18:55:37,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/B/e39d6e02d9b849aeb92e33be5fac44cc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/e39d6e02d9b849aeb92e33be5fac44cc 2023-04-05 18:55:37,254 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/e39d6e02d9b849aeb92e33be5fac44cc, entries=50, sequenceid=451, filesize=7.1 K 2023-04-05 18:55:37,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/.tmp/C/c2140b914c7c4f38bf6f7526d78ef18b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c2140b914c7c4f38bf6f7526d78ef18b 2023-04-05 18:55:37,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c2140b914c7c4f38bf6f7526d78ef18b, entries=50, sequenceid=451, filesize=7.1 K 2023-04-05 18:55:37,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 64f9b0847f90f48ecb0b352322029c8d in 1242ms, sequenceid=451, compaction requested=true 2023-04-05 18:55:37,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/1fba84a183884af788c5aef63a9b02e7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/9db6447a941e424badeae9aa7934b5c8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/f859841fedc84747b2c468d08f344fbb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/38e71152e6874b17b022907afbc008e5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/4f49e8e6b9794c7680bec77f06107aa8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/29318a358f04496fa9c4938fed7c47d0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/21d238494e16403caf06a3dfd0bb7ecb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/8d03101121c049128d8e9ea0b67cf1a9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/11a2930904e141f8ba50b02702e8c941, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b410e8eb01674ea2b9093c50ace9c805, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ae1fbfa0744a0596b2a4106e04e0bc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/eae144b56cf6435d96d49d21ea0d663f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d8dd451ce8de488697d2f2c67c9bad83, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/24b3adec06a149a9b8ce14ee442ce358, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b1625ceae4ae4cd696e638bb2e54f0ba, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ab193ad21640a3b1e9ae0ee3d37c52, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d6de259b7bd54cda86d88d537d1dba93, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/007626940db9483584a75dc7c20817c3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/05dd2d2a9fdc44f28500bda4eb037319, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/14d95cd0b8f24aed93c75520a46745fc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/0924e8ee688944b59421a3a1a773c013, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/2a03edb7b1bf41e594dad887e2a35fd0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/742389e3bb544122b7bc5d4b00626e58, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/5a8e823913714de1a7b4176f118484e0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/7357e976065e48c99b8517ad178e1688] to archive 2023-04-05 18:55:37,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:55:37,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/1fba84a183884af788c5aef63a9b02e7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/1fba84a183884af788c5aef63a9b02e7 2023-04-05 18:55:37,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/9db6447a941e424badeae9aa7934b5c8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/9db6447a941e424badeae9aa7934b5c8 2023-04-05 18:55:37,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/f859841fedc84747b2c468d08f344fbb to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/f859841fedc84747b2c468d08f344fbb 2023-04-05 18:55:37,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/38e71152e6874b17b022907afbc008e5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/38e71152e6874b17b022907afbc008e5 2023-04-05 18:55:37,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/4f49e8e6b9794c7680bec77f06107aa8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/4f49e8e6b9794c7680bec77f06107aa8 2023-04-05 18:55:37,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/29318a358f04496fa9c4938fed7c47d0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/29318a358f04496fa9c4938fed7c47d0 2023-04-05 18:55:37,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/21d238494e16403caf06a3dfd0bb7ecb to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/21d238494e16403caf06a3dfd0bb7ecb 2023-04-05 18:55:37,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/8d03101121c049128d8e9ea0b67cf1a9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/8d03101121c049128d8e9ea0b67cf1a9 2023-04-05 18:55:37,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/11a2930904e141f8ba50b02702e8c941 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/11a2930904e141f8ba50b02702e8c941 2023-04-05 18:55:37,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b410e8eb01674ea2b9093c50ace9c805 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b410e8eb01674ea2b9093c50ace9c805 2023-04-05 18:55:37,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ae1fbfa0744a0596b2a4106e04e0bc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ae1fbfa0744a0596b2a4106e04e0bc 2023-04-05 18:55:37,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/eae144b56cf6435d96d49d21ea0d663f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/eae144b56cf6435d96d49d21ea0d663f 2023-04-05 18:55:37,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d8dd451ce8de488697d2f2c67c9bad83 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d8dd451ce8de488697d2f2c67c9bad83 2023-04-05 18:55:37,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/24b3adec06a149a9b8ce14ee442ce358 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/24b3adec06a149a9b8ce14ee442ce358 2023-04-05 18:55:37,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b1625ceae4ae4cd696e638bb2e54f0ba to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b1625ceae4ae4cd696e638bb2e54f0ba 2023-04-05 18:55:37,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ab193ad21640a3b1e9ae0ee3d37c52 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/58ab193ad21640a3b1e9ae0ee3d37c52 2023-04-05 18:55:37,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d6de259b7bd54cda86d88d537d1dba93 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/d6de259b7bd54cda86d88d537d1dba93 2023-04-05 18:55:37,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/007626940db9483584a75dc7c20817c3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/007626940db9483584a75dc7c20817c3 2023-04-05 18:55:37,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/05dd2d2a9fdc44f28500bda4eb037319 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/05dd2d2a9fdc44f28500bda4eb037319 2023-04-05 18:55:37,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/14d95cd0b8f24aed93c75520a46745fc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/14d95cd0b8f24aed93c75520a46745fc 2023-04-05 18:55:37,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/0924e8ee688944b59421a3a1a773c013 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/0924e8ee688944b59421a3a1a773c013 2023-04-05 18:55:37,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/2a03edb7b1bf41e594dad887e2a35fd0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/2a03edb7b1bf41e594dad887e2a35fd0 2023-04-05 18:55:37,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/742389e3bb544122b7bc5d4b00626e58 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/742389e3bb544122b7bc5d4b00626e58 2023-04-05 18:55:37,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/5a8e823913714de1a7b4176f118484e0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/5a8e823913714de1a7b4176f118484e0 2023-04-05 18:55:37,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/7357e976065e48c99b8517ad178e1688 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/7357e976065e48c99b8517ad178e1688 2023-04-05 18:55:37,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/0c61fc8ed4b14a98b3a4e990b2170e51, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/30198d6b2be94408ab362aa03cbbff16, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/97500f9cda8848f28a4e6b4d9a1f7b07, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/7dda715ad7824fa4a7dac7515ec9368d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/69d50d3cee4d4492a0404f420dd7fd01, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/c517fdd8522545efb15be77d6034f599, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/fe569ac2825e4e54979573db391d1af1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/cbfb99f3f7a24f719ba88a39a01719d8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/01939b86d0bd410a9be4a84cc51cbd1e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0840ee9feae4d77a49a0cc474c587ce, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/639d40d27b5f4e95b5df805b93b5b114, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/14aa804ce87b4a799613ba9187ea9022, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/183fac1fcf5e45b9b46af29116b48f3e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/26feb215832e4187a706edee2b2cf35b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/67d7e5e76b8844edbfc70e460c945cc9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/036bbb39465f4ef1a66c083d63a00d59, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0ea0262925b43b2974705cb4ba6baf5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/278f6f3ed1c547fd8f0af7bd7565156c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/385892aa01ad4f78a683c58e635aaced, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/9b2a4e5d0c604dd08dada9874f972d35, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/794acd836fa942c9822258a6d1aea046, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/48830f20fc884764a26f0a92e7e0f45c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/2908abc1651048d7a6f2d63b2f232d3e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/177c1e55bae841d0b54c3372e4976d0e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/773b7255a01b415d850667246a7a3cd7] to archive 2023-04-05 18:55:37,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:55:37,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/0c61fc8ed4b14a98b3a4e990b2170e51 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/0c61fc8ed4b14a98b3a4e990b2170e51 2023-04-05 18:55:37,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/30198d6b2be94408ab362aa03cbbff16 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/30198d6b2be94408ab362aa03cbbff16 2023-04-05 18:55:37,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/97500f9cda8848f28a4e6b4d9a1f7b07 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/97500f9cda8848f28a4e6b4d9a1f7b07 2023-04-05 18:55:37,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/7dda715ad7824fa4a7dac7515ec9368d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/7dda715ad7824fa4a7dac7515ec9368d 2023-04-05 18:55:37,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/69d50d3cee4d4492a0404f420dd7fd01 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/69d50d3cee4d4492a0404f420dd7fd01 2023-04-05 18:55:37,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/c517fdd8522545efb15be77d6034f599 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/c517fdd8522545efb15be77d6034f599 2023-04-05 18:55:37,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/fe569ac2825e4e54979573db391d1af1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/fe569ac2825e4e54979573db391d1af1 2023-04-05 18:55:37,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/cbfb99f3f7a24f719ba88a39a01719d8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/cbfb99f3f7a24f719ba88a39a01719d8 2023-04-05 18:55:37,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/01939b86d0bd410a9be4a84cc51cbd1e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/01939b86d0bd410a9be4a84cc51cbd1e 2023-04-05 18:55:37,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0840ee9feae4d77a49a0cc474c587ce to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0840ee9feae4d77a49a0cc474c587ce 2023-04-05 18:55:37,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/639d40d27b5f4e95b5df805b93b5b114 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/639d40d27b5f4e95b5df805b93b5b114 2023-04-05 18:55:37,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/14aa804ce87b4a799613ba9187ea9022 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/14aa804ce87b4a799613ba9187ea9022 2023-04-05 18:55:37,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/183fac1fcf5e45b9b46af29116b48f3e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/183fac1fcf5e45b9b46af29116b48f3e 2023-04-05 18:55:37,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/26feb215832e4187a706edee2b2cf35b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/26feb215832e4187a706edee2b2cf35b 2023-04-05 18:55:37,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/67d7e5e76b8844edbfc70e460c945cc9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/67d7e5e76b8844edbfc70e460c945cc9 2023-04-05 18:55:37,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/036bbb39465f4ef1a66c083d63a00d59 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/036bbb39465f4ef1a66c083d63a00d59 2023-04-05 18:55:37,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0ea0262925b43b2974705cb4ba6baf5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/a0ea0262925b43b2974705cb4ba6baf5 2023-04-05 18:55:37,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/278f6f3ed1c547fd8f0af7bd7565156c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/278f6f3ed1c547fd8f0af7bd7565156c 2023-04-05 18:55:37,318 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/385892aa01ad4f78a683c58e635aaced to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/385892aa01ad4f78a683c58e635aaced 2023-04-05 18:55:37,318 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/9b2a4e5d0c604dd08dada9874f972d35 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/9b2a4e5d0c604dd08dada9874f972d35 2023-04-05 18:55:37,319 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/794acd836fa942c9822258a6d1aea046 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/794acd836fa942c9822258a6d1aea046 2023-04-05 18:55:37,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/48830f20fc884764a26f0a92e7e0f45c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/48830f20fc884764a26f0a92e7e0f45c 2023-04-05 18:55:37,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/2908abc1651048d7a6f2d63b2f232d3e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/2908abc1651048d7a6f2d63b2f232d3e 2023-04-05 18:55:37,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/177c1e55bae841d0b54c3372e4976d0e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/177c1e55bae841d0b54c3372e4976d0e 2023-04-05 18:55:37,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/773b7255a01b415d850667246a7a3cd7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/773b7255a01b415d850667246a7a3cd7 2023-04-05 18:55:37,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/68ac5559e9ae48c3a957bdedd2657c20, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/83562c1ff5ea47df8d287f6745f19b3f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77714b42ab874ae8a54c763f10a4453d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/008b7b5b46e741d1be78fad663e5b6fe, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f4ac4e2d39ee437e806edb71af5e406f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c056ceb0a53644e983cd2fcb754a5a11, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/288490062f8a467e8b633aab9ae0ee2f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/5712a0fa5d764ec1a8d5224abcd0b4b7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c7bf5f7237d540aeb386f866b76e5f97, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1f7aa6b1937c488fb711096b8637f225, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/005fcd8da3db4267a80a867902e1d5f8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/ede5786254f146ec8e01dd01a44b7eca, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/cd9de5a7d99745bc89c3ac40ba703f93, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77825564ca58455aa7bb5c7cfc7bac33, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/45fc5d86089e4c1597bda41f3d25d307, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f6169657d27f439f8c380c7269600d93, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/0fc175f1c8ec44c5a93716cf955b2c38, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1bd6998550e943ad8f07b9496c6b35da, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/a533cf366ca74fb6b5a3673e4dbab725, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/959e779706a241dcb395fa0f2d6f34e6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/b94b1fd549d44c109f75e854b41a0a1d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/7fcc4441f5ff4b468cb6f919b77adb88, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/fe5998047235427baaa04938197f9c32, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/58fb790acd624960b5f9e269deec1f5e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/8c32d5ea677e4d9ebfc0451a9e44e4f1] to archive 2023-04-05 18:55:37,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:55:37,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/68ac5559e9ae48c3a957bdedd2657c20 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/68ac5559e9ae48c3a957bdedd2657c20 2023-04-05 18:55:37,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/83562c1ff5ea47df8d287f6745f19b3f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/83562c1ff5ea47df8d287f6745f19b3f 2023-04-05 18:55:37,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77714b42ab874ae8a54c763f10a4453d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77714b42ab874ae8a54c763f10a4453d 2023-04-05 18:55:37,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/008b7b5b46e741d1be78fad663e5b6fe to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/008b7b5b46e741d1be78fad663e5b6fe 2023-04-05 18:55:37,329 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f4ac4e2d39ee437e806edb71af5e406f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f4ac4e2d39ee437e806edb71af5e406f 2023-04-05 18:55:37,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c056ceb0a53644e983cd2fcb754a5a11 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c056ceb0a53644e983cd2fcb754a5a11 2023-04-05 18:55:37,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/288490062f8a467e8b633aab9ae0ee2f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/288490062f8a467e8b633aab9ae0ee2f 2023-04-05 18:55:37,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/5712a0fa5d764ec1a8d5224abcd0b4b7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/5712a0fa5d764ec1a8d5224abcd0b4b7 2023-04-05 18:55:37,332 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c7bf5f7237d540aeb386f866b76e5f97 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c7bf5f7237d540aeb386f866b76e5f97 2023-04-05 18:55:37,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1f7aa6b1937c488fb711096b8637f225 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1f7aa6b1937c488fb711096b8637f225 2023-04-05 18:55:37,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/005fcd8da3db4267a80a867902e1d5f8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/005fcd8da3db4267a80a867902e1d5f8 2023-04-05 18:55:37,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/ede5786254f146ec8e01dd01a44b7eca to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/ede5786254f146ec8e01dd01a44b7eca 2023-04-05 18:55:37,335 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/cd9de5a7d99745bc89c3ac40ba703f93 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/cd9de5a7d99745bc89c3ac40ba703f93 2023-04-05 18:55:37,336 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77825564ca58455aa7bb5c7cfc7bac33 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/77825564ca58455aa7bb5c7cfc7bac33 2023-04-05 18:55:37,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/45fc5d86089e4c1597bda41f3d25d307 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/45fc5d86089e4c1597bda41f3d25d307 2023-04-05 18:55:37,338 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f6169657d27f439f8c380c7269600d93 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/f6169657d27f439f8c380c7269600d93 2023-04-05 18:55:37,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/0fc175f1c8ec44c5a93716cf955b2c38 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/0fc175f1c8ec44c5a93716cf955b2c38 2023-04-05 18:55:37,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1bd6998550e943ad8f07b9496c6b35da to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/1bd6998550e943ad8f07b9496c6b35da 2023-04-05 18:55:37,340 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/a533cf366ca74fb6b5a3673e4dbab725 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/a533cf366ca74fb6b5a3673e4dbab725 2023-04-05 18:55:37,341 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/959e779706a241dcb395fa0f2d6f34e6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/959e779706a241dcb395fa0f2d6f34e6 2023-04-05 18:55:37,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/b94b1fd549d44c109f75e854b41a0a1d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/b94b1fd549d44c109f75e854b41a0a1d 2023-04-05 18:55:37,343 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/7fcc4441f5ff4b468cb6f919b77adb88 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/7fcc4441f5ff4b468cb6f919b77adb88 2023-04-05 18:55:37,343 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/fe5998047235427baaa04938197f9c32 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/fe5998047235427baaa04938197f9c32 2023-04-05 18:55:37,344 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/58fb790acd624960b5f9e269deec1f5e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/58fb790acd624960b5f9e269deec1f5e 2023-04-05 18:55:37,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/8c32d5ea677e4d9ebfc0451a9e44e4f1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/8c32d5ea677e4d9ebfc0451a9e44e4f1 2023-04-05 18:55:37,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/recovered.edits/454.seqid, newMaxSeqId=454, maxSeqId=4 2023-04-05 18:55:37,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d. 2023-04-05 18:55:37,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 64f9b0847f90f48ecb0b352322029c8d: 2023-04-05 18:55:37,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:37,351 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=64f9b0847f90f48ecb0b352322029c8d, regionState=CLOSED 2023-04-05 18:55:37,351 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720937351"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720937351"}]},"ts":"1680720937351"} 2023-04-05 18:55:37,353 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-05 18:55:37,353 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 64f9b0847f90f48ecb0b352322029c8d, server=jenkins-hbase4.apache.org,33489,1680720816595 in 1.4880 sec 2023-04-05 18:55:37,354 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-05 18:55:37,354 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=64f9b0847f90f48ecb0b352322029c8d, UNASSIGN in 1.4910 sec 2023-04-05 18:55:37,354 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720937354"}]},"ts":"1680720937354"} 2023-04-05 18:55:37,355 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 18:55:37,356 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 18:55:37,357 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5020 sec 2023-04-05 18:55:37,811 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-05 18:55:37,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 18:55:37,963 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-05 18:55:37,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 18:55:37,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:37,965 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:37,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-05 18:55:37,966 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:37,969 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:37,971 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/recovered.edits] 2023-04-05 18:55:37,973 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/43060306d31f442db1177bd100f8ad1a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/43060306d31f442db1177bd100f8ad1a 2023-04-05 18:55:37,975 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/7eabf80323164bb284a7976a7f764eac to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/7eabf80323164bb284a7976a7f764eac 2023-04-05 18:55:37,976 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b2e74cde43e64bf393483fd7cd828166 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/A/b2e74cde43e64bf393483fd7cd828166 2023-04-05 18:55:37,977 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/2edaa0cee5914d0ca904e784bcae1ef5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/2edaa0cee5914d0ca904e784bcae1ef5 2023-04-05 18:55:37,978 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/c5226bdd4e9647adae6e7cf378b64e92 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/c5226bdd4e9647adae6e7cf378b64e92 2023-04-05 18:55:37,979 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/e39d6e02d9b849aeb92e33be5fac44cc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/B/e39d6e02d9b849aeb92e33be5fac44cc 2023-04-05 18:55:37,981 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/9fd79770be8944349393115ac73ccb3c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/9fd79770be8944349393115ac73ccb3c 2023-04-05 18:55:37,982 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/be2378e164ec4db5acb1a92931a1b150 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/be2378e164ec4db5acb1a92931a1b150 2023-04-05 18:55:37,983 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c2140b914c7c4f38bf6f7526d78ef18b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/C/c2140b914c7c4f38bf6f7526d78ef18b 2023-04-05 18:55:37,985 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/recovered.edits/454.seqid to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d/recovered.edits/454.seqid 2023-04-05 18:55:37,985 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/64f9b0847f90f48ecb0b352322029c8d 2023-04-05 18:55:37,985 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:55:37,985 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 18:55:37,986 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-05 18:55:37,988 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050c31a714c9734f18b24f8bf6b8e4f6eb to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304050c31a714c9734f18b24f8bf6b8e4f6eb 2023-04-05 18:55:37,989 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051f643721daa6403689a171c917c4b8b4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051f643721daa6403689a171c917c4b8b4 2023-04-05 18:55:37,990 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053b409aadd6164d909ad9d756a4bc3ae7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053b409aadd6164d909ad9d756a4bc3ae7 2023-04-05 18:55:37,991 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053c0cfca071c3430a8d26dd1a7447d35d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053c0cfca071c3430a8d26dd1a7447d35d 2023-04-05 18:55:37,991 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053c482cc05b684bca84f143e7c88b44c6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053c482cc05b684bca84f143e7c88b44c6 2023-04-05 18:55:37,992 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405470887fc699e4f26ba6b97c27d49a33e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405470887fc699e4f26ba6b97c27d49a33e 2023-04-05 18:55:37,993 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405488fef021e8a4af0838eaed146f26bcc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405488fef021e8a4af0838eaed146f26bcc 2023-04-05 18:55:37,994 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054e9757b26f204d90b1ce84ad52ccae17 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054e9757b26f204d90b1ce84ad52ccae17 2023-04-05 18:55:37,995 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055491c2d0fab2475b802f90c0775b89a2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055491c2d0fab2475b802f90c0775b89a2 2023-04-05 18:55:37,996 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055833e9a7a1ca4257a5134e7858b7d0e9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055833e9a7a1ca4257a5134e7858b7d0e9 2023-04-05 18:55:37,997 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405659d17eaf0ef4990a86e2ad6d6d0fd09 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405659d17eaf0ef4990a86e2ad6d6d0fd09 2023-04-05 18:55:37,998 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040576b743bbf82e484cb17be1537a5e48a5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040576b743bbf82e484cb17be1537a5e48a5 2023-04-05 18:55:37,999 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040579f291949ae943f7a08c7793f77ee33f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040579f291949ae943f7a08c7793f77ee33f 2023-04-05 18:55:38,000 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b6c3053a9ddf4a7d897f7cf258b82897 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b6c3053a9ddf4a7d897f7cf258b82897 2023-04-05 18:55:38,001 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bfcb689b53f24928a9aa7f1222b56aae to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bfcb689b53f24928a9aa7f1222b56aae 2023-04-05 18:55:38,002 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d69a867a2d5b4d44b3bd24900fafc3a2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d69a867a2d5b4d44b3bd24900fafc3a2 2023-04-05 18:55:38,002 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d6d182c10a2b4a869e2b8ea6505eb436 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d6d182c10a2b4a869e2b8ea6505eb436 2023-04-05 18:55:38,003 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d965157df3f84a1e86b055dfe7937175 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d965157df3f84a1e86b055dfe7937175 2023-04-05 18:55:38,004 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e232b55315124e3d9b233a76512e4da4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e232b55315124e3d9b233a76512e4da4 2023-04-05 18:55:38,005 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f2a1f168ae844676bbcf4b3052bde00a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405f2a1f168ae844676bbcf4b3052bde00a 2023-04-05 18:55:38,006 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405fb67ac22c7934a489e1c365e3a90e06a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405fb67ac22c7934a489e1c365e3a90e06a 2023-04-05 18:55:38,006 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ff96a5b1ed5b4e6c8e8427713f6c86f7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ff96a5b1ed5b4e6c8e8427713f6c86f7 2023-04-05 18:55:38,007 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 18:55:38,009 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:38,010 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 18:55:38,012 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 18:55:38,013 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:38,013 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 18:55:38,013 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680720938013"}]},"ts":"9223372036854775807"} 2023-04-05 18:55:38,015 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 18:55:38,015 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 64f9b0847f90f48ecb0b352322029c8d, NAME => 'TestAcidGuarantees,,1680720908443.64f9b0847f90f48ecb0b352322029c8d.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 18:55:38,015 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 18:55:38,015 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680720938015"}]},"ts":"9223372036854775807"} 2023-04-05 18:55:38,019 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 18:55:38,021 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:38,022 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 58 msec 2023-04-05 18:55:38,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-05 18:55:38,067 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-05 18:55:38,079 INFO [Listener at localhost/43705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 227), OpenFileDescriptor=539 (was 555), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 413), ProcessCount=169 (was 169), AvailableMemoryMB=1178 (was 1299) 2023-04-05 18:55:38,093 INFO [Listener at localhost/43705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=539, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=169, AvailableMemoryMB=1178 2023-04-05 18:55:38,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 18:55:38,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 18:55:38,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 18:55:38,097 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 18:55:38,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-05 18:55:38,097 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 18:55:38,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 18:55:38,098 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,099 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010 empty. 2023-04-05 18:55:38,099 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,099 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:55:38,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 18:55:38,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 18:55:38,506 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 18:55:38,507 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6e819dddd14a04b6eb4e71fe4e02a010, NAME => 'TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp 2023-04-05 18:55:38,516 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:55:38,516 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6e819dddd14a04b6eb4e71fe4e02a010, disabling compactions & flushes 2023-04-05 18:55:38,516 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:38,516 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:38,516 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. after waiting 0 ms 2023-04-05 18:55:38,517 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:38,517 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:38,517 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:38,518 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 18:55:38,519 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720938519"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720938519"}]},"ts":"1680720938519"} 2023-04-05 18:55:38,520 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 18:55:38,521 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 18:55:38,521 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720938521"}]},"ts":"1680720938521"} 2023-04-05 18:55:38,522 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 18:55:38,527 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6e819dddd14a04b6eb4e71fe4e02a010, ASSIGN}] 2023-04-05 18:55:38,529 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6e819dddd14a04b6eb4e71fe4e02a010, ASSIGN 2023-04-05 18:55:38,529 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6e819dddd14a04b6eb4e71fe4e02a010, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33489,1680720816595; forceNewPlan=false, retain=false 2023-04-05 18:55:38,681 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=6e819dddd14a04b6eb4e71fe4e02a010, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:38,681 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720938680"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720938680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720938680"}]},"ts":"1680720938680"} 2023-04-05 18:55:38,682 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:55:38,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 18:55:38,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:38,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6e819dddd14a04b6eb4e71fe4e02a010, NAME => 'TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:55:38,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:55:38,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,838 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,839 DEBUG [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A 2023-04-05 18:55:38,839 DEBUG [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A 2023-04-05 18:55:38,839 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:55:38,839 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6e819dddd14a04b6eb4e71fe4e02a010 columnFamilyName A 2023-04-05 18:55:38,840 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] regionserver.HStore(310): Store=6e819dddd14a04b6eb4e71fe4e02a010/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:55:38,840 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,841 DEBUG [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B 2023-04-05 18:55:38,841 DEBUG [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B 2023-04-05 18:55:38,841 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:55:38,842 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6e819dddd14a04b6eb4e71fe4e02a010 columnFamilyName B 2023-04-05 18:55:38,842 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] regionserver.HStore(310): Store=6e819dddd14a04b6eb4e71fe4e02a010/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:55:38,842 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,843 DEBUG [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C 2023-04-05 18:55:38,843 DEBUG [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C 2023-04-05 18:55:38,843 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:55:38,843 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6e819dddd14a04b6eb4e71fe4e02a010 columnFamilyName C 2023-04-05 18:55:38,844 INFO [StoreOpener-6e819dddd14a04b6eb4e71fe4e02a010-1] regionserver.HStore(310): Store=6e819dddd14a04b6eb4e71fe4e02a010/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:55:38,844 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:38,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:55:38,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:38,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 18:55:38,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6e819dddd14a04b6eb4e71fe4e02a010; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73047946, jitterRate=0.08849921822547913}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:55:38,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:38,850 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., pid=49, masterSystemTime=1680720938833 2023-04-05 18:55:38,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:38,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:38,851 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=6e819dddd14a04b6eb4e71fe4e02a010, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:38,851 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720938851"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680720938851"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680720938851"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680720938851"}]},"ts":"1680720938851"} 2023-04-05 18:55:38,853 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-05 18:55:38,853 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 in 170 msec 2023-04-05 18:55:38,854 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-05 18:55:38,854 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6e819dddd14a04b6eb4e71fe4e02a010, ASSIGN in 326 msec 2023-04-05 18:55:38,854 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 18:55:38,855 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720938854"}]},"ts":"1680720938854"} 2023-04-05 18:55:38,855 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 18:55:38,857 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 18:55:38,858 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 762 msec 2023-04-05 18:55:39,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 18:55:39,201 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-05 18:55:39,202 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x37705401 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,206 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec943a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,207 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,209 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41994, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,210 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 18:55:39,212 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 18:55:39,213 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,218 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,218 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,223 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,223 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x06edd017 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,227 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51b8df2a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,228 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x0fb168ce to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,234 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67346c47, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,235 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x461f5f3f to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,238 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@426cb82a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,238 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x5f5363d8 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,243 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78592df0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,243 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x211582f2 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,250 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@659c68ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,250 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x65816b6e to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,254 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@680e773e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,254 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x653bb8fc to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,262 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b8f652e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,262 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x720d29f5 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:55:39,267 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ac98310, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:55:39,276 DEBUG [hconnection-0x589f3651-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,281 DEBUG [hconnection-0x55994f13-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,282 DEBUG [hconnection-0x4814f6e1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,282 DEBUG [hconnection-0x51bdb15d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,282 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42020, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,285 DEBUG [hconnection-0x630f9006-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,285 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42032, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,286 DEBUG [hconnection-0x24c5c8c1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,286 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42062, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:39,290 DEBUG [hconnection-0x16c51410-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,290 DEBUG [hconnection-0x56182a06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,291 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,294 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,296 DEBUG [hconnection-0x65529ca6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,296 DEBUG [hconnection-0xdfcca3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:55:39,297 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42008, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,297 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42048, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,297 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,303 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,303 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42114, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:55:39,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:39,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:39,311 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:39,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:39,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:39,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:55:39,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:39,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:39,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,313 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:39,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:39,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:39,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:39,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:39,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:39,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:39,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:39,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:39,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:39,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:39,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:39,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:39,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:39,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:39,324 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:39,324 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:39,324 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:39,324 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:39,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,324 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:39,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680720999323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,324 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:39,324 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:39,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:39,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:39,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:39,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680720999323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680720939322, max timestamp=1680720939323Num uniques 859; 2023-04-05 18:55:39,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680720939320, max timestamp=1680720939323Num uniques 859; 2023-04-05 18:55:39,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:39,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680720939322, max timestamp=1680720939323Num uniques 859; 2023-04-05 18:55:39,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:55:39,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:39,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,327 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,327 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,327 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:39,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@599acd7e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:39,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:39,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,330 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:39,330 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,330 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:39,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:39,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:39,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680720999333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680720999334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680720999334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,335 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,335 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:39,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,335 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:39,335 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:39,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:39,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:39,338 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:39,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:39,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:39,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:39,338 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:39,339 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:39,340 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:39,340 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,342 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:39,344 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:39,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:39,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:39,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:39,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:39,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:39,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:39,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:39,345 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:39,345 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:39,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680720999426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680720999427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680720999434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680720999435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680720999435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,445 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:39,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:39,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:39,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:39,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:39,456 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:39,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:39,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:39,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:39,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,457 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,458 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:39,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:39,458 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,458 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:39,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:39,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:39,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:39,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:39,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:39,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:39,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:39,465 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:39,465 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:39,465 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:39,465 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:39,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:39,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:39,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:39,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:39,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,467 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:39,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f79160[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:39,469 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:39,469 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,471 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:39,471 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:39,471 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:39,471 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,471 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:39,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:39,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:39,475 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:39,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:39,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:39,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:39,475 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:39,477 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,477 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,477 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,478 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:39,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,478 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,484 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,484 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,484 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,484 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,484 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,484 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,484 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:39,485 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:39,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:39,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:39,485 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:39,485 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:39,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:39,486 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:39,486 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:39,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:39,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:39,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:39,586 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:39,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:39,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:39,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:39,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:39,607 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:39,607 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:39,607 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:39,607 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:39,608 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,608 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,609 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:39,609 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:39,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:39,609 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,609 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:39,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:39,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:39,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:39,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:39,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:39,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:39,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:39,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:39,621 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:39,622 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:39,622 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:39,622 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:39,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:39,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:39,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:39,622 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:39,623 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,623 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:39,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79250845[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:39,626 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:39,626 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,627 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:39,627 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:39,627 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:39,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:39,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680720999628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680720999629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,630 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:39,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:39,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:39,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:39,635 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:39,635 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:39,635 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:39,635 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:39,635 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:39,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680720999638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:39,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,639 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:39,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680720999639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,640 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,641 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:39,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:39,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:39,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:39,644 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:39,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:39,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:39,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:39,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:39,644 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:39,644 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:39,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680720999644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,645 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/2c3afd2982114da690d89f1669d2fbdb 2023-04-05 18:55:39,745 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:39,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:39,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:39,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:39,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:39,761 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:39,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:39,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:39,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:39,762 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,763 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,765 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,765 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:39,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:39,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:39,765 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,765 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:39,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:39,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:39,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:39,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:39,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:39,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:39,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:39,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:39,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:39,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:39,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:39,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:39,777 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:39,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:39,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:39,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,780 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:39,780 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:39,782 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,782 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,783 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:39,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:39,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e84e146[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:39,784 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:39,784 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,786 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:39,786 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:39,786 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:39,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:39,787 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:39,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:39,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:39,790 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:39,790 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:39,790 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:39,790 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:39,790 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:39,793 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,794 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,794 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:39,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:39,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:39,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:39,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,799 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,802 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:39,802 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:39,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:39,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:39,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:39,803 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:39,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:39,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:39,804 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:39,804 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:39,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/f62e4bcb945c4336a3a595b1d41ffe2f 2023-04-05 18:55:39,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/4456fe4c1c8449a0b835b8d94b308f16 2023-04-05 18:55:39,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/2c3afd2982114da690d89f1669d2fbdb as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2c3afd2982114da690d89f1669d2fbdb 2023-04-05 18:55:39,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2c3afd2982114da690d89f1669d2fbdb, entries=150, sequenceid=14, filesize=11.6 K 2023-04-05 18:55:39,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/f62e4bcb945c4336a3a595b1d41ffe2f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f62e4bcb945c4336a3a595b1d41ffe2f 2023-04-05 18:55:39,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f62e4bcb945c4336a3a595b1d41ffe2f, entries=150, sequenceid=14, filesize=11.6 K 2023-04-05 18:55:39,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/4456fe4c1c8449a0b835b8d94b308f16 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4456fe4c1c8449a0b835b8d94b308f16 2023-04-05 18:55:39,854 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4456fe4c1c8449a0b835b8d94b308f16, entries=150, sequenceid=14, filesize=11.6 K 2023-04-05 18:55:39,855 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=116.29 KB/119080 for 6e819dddd14a04b6eb4e71fe4e02a010 in 544ms, sequenceid=14, compaction requested=false 2023-04-05 18:55:39,855 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-05 18:55:39,861 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:39,904 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:39,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:39,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:39,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:39,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:39,910 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:39,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:39,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:39,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:39,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,911 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:39,911 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:39,912 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,912 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:39,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:39,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:39,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:39,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:39,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:39,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:39,913 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:39,913 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:39,913 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:39,913 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:39,913 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:39,913 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:39,913 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:39,913 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=116.29 KB heapSize=325.61 KB 2023-04-05 18:55:39,914 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:39,914 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:39,914 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:39,914 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:39,914 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:39,914 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:39,934 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/eabeb98b45d9420f89b07c12dcd311aa 2023-04-05 18:55:39,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:39,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:39,949 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/e9700cfe63a847589fe12f1f990a9efa 2023-04-05 18:55:39,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680720999946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680720999948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680720999949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:39,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680720999959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:39,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680720999959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721000060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721000061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721000061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721000071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721000071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721000264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721000267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721000267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721000277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721000278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,371 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/0ff12166cf9b4f6a8ac47bf9f0ff8804 2023-04-05 18:55:40,375 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/eabeb98b45d9420f89b07c12dcd311aa as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/eabeb98b45d9420f89b07c12dcd311aa 2023-04-05 18:55:40,381 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/eabeb98b45d9420f89b07c12dcd311aa, entries=150, sequenceid=47, filesize=11.6 K 2023-04-05 18:55:40,382 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/e9700cfe63a847589fe12f1f990a9efa as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/e9700cfe63a847589fe12f1f990a9efa 2023-04-05 18:55:40,385 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/e9700cfe63a847589fe12f1f990a9efa, entries=150, sequenceid=47, filesize=11.6 K 2023-04-05 18:55:40,385 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/0ff12166cf9b4f6a8ac47bf9f0ff8804 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0ff12166cf9b4f6a8ac47bf9f0ff8804 2023-04-05 18:55:40,388 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0ff12166cf9b4f6a8ac47bf9f0ff8804, entries=150, sequenceid=47, filesize=11.6 K 2023-04-05 18:55:40,388 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~116.29 KB/119080, heapSize ~325.56 KB/333376, currentSize=67.09 KB/68700 for 6e819dddd14a04b6eb4e71fe4e02a010 in 475ms, sequenceid=47, compaction requested=false 2023-04-05 18:55:40,390 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:40,390 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:40,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:40,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:40,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:40,390 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:40,391 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,391 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:40,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:40,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51fa26d5[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:40,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:40,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:40,395 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:40,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:40,395 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,395 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:40,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:40,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:40,399 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:40,399 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:40,399 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:40,399 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:40,399 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:40,400 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:40,400 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:40,401 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:40,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:40,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:40,401 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:40,402 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,404 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:40,404 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:40,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:40,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:40,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:40,404 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,405 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:40,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:40,404 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:40,405 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:40,405 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:40,405 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:40,405 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,405 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:40,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:40,405 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:40,405 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:40,505 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:40,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:40,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:40,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:40,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:40,508 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:40,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:40,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:40,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:40,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:40,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:40,511 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:40,511 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:40,511 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,511 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:40,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:40,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:40,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:40,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:40,512 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:40,512 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:40,513 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:40,513 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:40,513 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:40,513 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:40,513 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:40,513 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 18:55:40,513 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:40,513 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:40,513 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:40,513 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:40,513 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:40,513 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:40,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:40,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:40,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721000593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721000596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721000600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721000601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721000602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721000703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721000703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721000707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721000710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721000710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721000908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721000908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721000911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721000915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:40,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721000915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:40,922 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/208dbed4bd8747808c5fc3ae9f3414f8 2023-04-05 18:55:40,933 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/983ef6ecd07f40b6901c4eea05f35870 2023-04-05 18:55:41,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721001212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721001212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721001221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721001221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721001223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,347 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/ab7cdb51e4964425b38d5792f8540da0 2023-04-05 18:55:41,351 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/208dbed4bd8747808c5fc3ae9f3414f8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/208dbed4bd8747808c5fc3ae9f3414f8 2023-04-05 18:55:41,354 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/208dbed4bd8747808c5fc3ae9f3414f8, entries=150, sequenceid=60, filesize=11.6 K 2023-04-05 18:55:41,355 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/983ef6ecd07f40b6901c4eea05f35870 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/983ef6ecd07f40b6901c4eea05f35870 2023-04-05 18:55:41,358 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/983ef6ecd07f40b6901c4eea05f35870, entries=150, sequenceid=60, filesize=11.6 K 2023-04-05 18:55:41,359 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/ab7cdb51e4964425b38d5792f8540da0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/ab7cdb51e4964425b38d5792f8540da0 2023-04-05 18:55:41,362 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/ab7cdb51e4964425b38d5792f8540da0, entries=150, sequenceid=60, filesize=11.6 K 2023-04-05 18:55:41,362 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6e819dddd14a04b6eb4e71fe4e02a010 in 849ms, sequenceid=60, compaction requested=true 2023-04-05 18:55:41,362 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:41,363 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:41,363 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:41,363 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:41,363 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,363 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:41,363 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:41,364 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,364 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:41,364 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:41,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:41,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:41,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35e2d11b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:41,366 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:41,366 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:41,367 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:41,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:41,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:41,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:41,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:41,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:41,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:41,368 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:41,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:41,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:41,368 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:41,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:41,370 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:41,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:41,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:41,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:41,370 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:41,373 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:41,373 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:41,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:41,373 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:41,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:41,373 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:41,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:41,375 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,379 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:41,379 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:41,379 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:41,379 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:41,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:41,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:41,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:41,380 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:41,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:41,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:41,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:41,380 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:41,380 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:41,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:41,480 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:41,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:41,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:41,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:41,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:41,483 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:41,484 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:41,484 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:41,484 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:41,484 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:41,484 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:41,485 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,485 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:41,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:41,485 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,485 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:41,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:41,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:41,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:41,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:41,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:41,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:41,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:41,487 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:41,487 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:41,487 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:41,487 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:41,487 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 18:55:41,487 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:41,487 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:41,487 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:41,487 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:41,487 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:41,487 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:41,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:41,728 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:41,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721001737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721001738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721001739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721001744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721001745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721001846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721001846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721001846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721001852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:41,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721001853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:41,895 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/d5ac58e007d34f7aa0343e55f327f2ba 2023-04-05 18:55:42,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721002050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721002050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721002051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721002057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721002058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,305 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/b712f544f53943baa19796add81929ac 2023-04-05 18:55:42,315 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/928052bb087a435dafdd44c505dc0e32 2023-04-05 18:55:42,324 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/d5ac58e007d34f7aa0343e55f327f2ba as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d5ac58e007d34f7aa0343e55f327f2ba 2023-04-05 18:55:42,337 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d5ac58e007d34f7aa0343e55f327f2ba, entries=150, sequenceid=79, filesize=11.6 K 2023-04-05 18:55:42,338 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/b712f544f53943baa19796add81929ac as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b712f544f53943baa19796add81929ac 2023-04-05 18:55:42,342 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b712f544f53943baa19796add81929ac, entries=150, sequenceid=79, filesize=11.6 K 2023-04-05 18:55:42,344 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/928052bb087a435dafdd44c505dc0e32 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/928052bb087a435dafdd44c505dc0e32 2023-04-05 18:55:42,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721002357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721002357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721002358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721002363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721002366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,375 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/928052bb087a435dafdd44c505dc0e32, entries=150, sequenceid=79, filesize=11.6 K 2023-04-05 18:55:42,376 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6e819dddd14a04b6eb4e71fe4e02a010 in 889ms, sequenceid=79, compaction requested=true 2023-04-05 18:55:42,376 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:42,377 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:42,377 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:42,377 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:42,377 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,377 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:42,377 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:42,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,380 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:42,380 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:42,380 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:42,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:42,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645480c2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:42,382 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:42,382 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:42,384 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:42,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:42,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:42,384 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:42,384 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:42,384 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:42,384 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:42,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:42,387 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:42,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:42,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:42,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:42,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:42,391 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:42,391 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:42,391 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:42,391 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:42,391 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:42,393 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:42,393 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:42,393 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:42,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:42,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:42,394 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:42,395 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:42,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:42,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,399 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:42,399 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:42,399 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:42,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:42,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:42,399 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:42,399 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:42,399 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,399 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:42,399 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:42,399 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:42,399 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:42,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:42,399 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:42,399 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:42,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:42,499 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:42,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:42,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:42,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:42,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:42,502 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:42,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:42,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:42,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:42,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:42,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:42,504 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:42,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:42,505 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,505 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:42,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:42,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:42,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:42,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:42,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:42,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:42,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:42,506 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:42,506 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:42,506 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:42,506 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:42,506 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:42,506 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:42,506 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:42,506 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 18:55:42,506 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:42,506 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:42,507 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:42,507 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:42,507 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:42,507 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:42,890 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 18:55:42,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:42,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:42,921 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/56b718db957f49199b9c128414f5ca0f 2023-04-05 18:55:42,934 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/57f6a20fa19a400dbf533819520d9bc7 2023-04-05 18:55:42,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721002931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721002931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721002932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721002933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:42,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:42,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721002935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721003041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721003041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721003042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721003042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721003043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721003249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721003249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721003249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721003249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721003250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,350 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/32d69cda5ca945c8a44fde188ebe7cd5 2023-04-05 18:55:43,354 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/56b718db957f49199b9c128414f5ca0f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/56b718db957f49199b9c128414f5ca0f 2023-04-05 18:55:43,357 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/56b718db957f49199b9c128414f5ca0f, entries=150, sequenceid=92, filesize=11.6 K 2023-04-05 18:55:43,358 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/57f6a20fa19a400dbf533819520d9bc7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/57f6a20fa19a400dbf533819520d9bc7 2023-04-05 18:55:43,361 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/57f6a20fa19a400dbf533819520d9bc7, entries=150, sequenceid=92, filesize=11.6 K 2023-04-05 18:55:43,362 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/32d69cda5ca945c8a44fde188ebe7cd5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32d69cda5ca945c8a44fde188ebe7cd5 2023-04-05 18:55:43,365 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32d69cda5ca945c8a44fde188ebe7cd5, entries=150, sequenceid=92, filesize=11.6 K 2023-04-05 18:55:43,365 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6e819dddd14a04b6eb4e71fe4e02a010 in 859ms, sequenceid=92, compaction requested=true 2023-04-05 18:55:43,365 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:43,365 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:43,366 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:43,366 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:43,366 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,366 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:43,366 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:43,367 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da3d3df[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:43,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:43,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,370 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:43,370 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:43,370 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:43,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:43,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:43,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:43,373 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,373 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:43,375 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:43,375 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:43,375 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:43,375 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:43,375 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:43,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:43,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:43,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:43,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,378 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:43,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:43,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:43,381 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:43,380 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:43,381 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:43,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:43,381 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:43,381 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:43,381 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,381 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,381 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:43,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:43,481 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:43,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:43,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:43,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:43,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:43,484 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:43,484 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:43,484 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:43,485 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:43,485 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,485 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:43,486 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:43,486 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:43,486 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,486 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:43,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:43,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:43,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:43,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:43,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:43,487 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:43,487 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:43,487 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:43,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:43,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:43,488 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 18:55:43,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:43,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:43,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:43,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:43,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:43,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:43,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:43,561 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:43,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721003573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721003574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721003575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721003575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721003575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721003676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721003682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721003683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721003683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721003683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721003883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721003891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721003891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721003892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721003892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,897 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/d9bf50cd79a24b3a9cc69dba8eb28d25 2023-04-05 18:55:43,906 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/00c3dd3121b14b5f8ae954d377dc8514 2023-04-05 18:55:43,916 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/0819993c38ef497abcc238bb9d449fb9 2023-04-05 18:55:43,919 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/d9bf50cd79a24b3a9cc69dba8eb28d25 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d9bf50cd79a24b3a9cc69dba8eb28d25 2023-04-05 18:55:43,922 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d9bf50cd79a24b3a9cc69dba8eb28d25, entries=150, sequenceid=111, filesize=11.6 K 2023-04-05 18:55:43,923 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/00c3dd3121b14b5f8ae954d377dc8514 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/00c3dd3121b14b5f8ae954d377dc8514 2023-04-05 18:55:43,925 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/00c3dd3121b14b5f8ae954d377dc8514, entries=150, sequenceid=111, filesize=11.6 K 2023-04-05 18:55:43,926 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/0819993c38ef497abcc238bb9d449fb9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0819993c38ef497abcc238bb9d449fb9 2023-04-05 18:55:43,929 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0819993c38ef497abcc238bb9d449fb9, entries=150, sequenceid=111, filesize=11.6 K 2023-04-05 18:55:43,930 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 6e819dddd14a04b6eb4e71fe4e02a010 in 441ms, sequenceid=111, compaction requested=true 2023-04-05 18:55:43,930 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:43,930 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:43,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:43,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:43,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:43,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:43,932 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:43,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:43,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d6f9230[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:43,934 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:43,934 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,934 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:43,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:43,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:43,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:43,936 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:43,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:43,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:43,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:43,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:43,940 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:43,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:43,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:43,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:43,940 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:43,941 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,941 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:43,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,941 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:43,942 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:43,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:43,943 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:43,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:43,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,947 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,947 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:43,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:43,947 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:43,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:43,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:43,947 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:43,948 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:43,948 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,948 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:43,948 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:43,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:43,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:43,947 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:43,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:43,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:43,948 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:43,948 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:44,048 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:44,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:44,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:44,051 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:44,051 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:44,051 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:44,052 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:44,052 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:44,052 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:44,056 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:44,056 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:44,056 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,056 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:44,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:44,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:44,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:44,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:44,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:44,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:44,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:44,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:44,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:44,059 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 18:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:44,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:44,195 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:44,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721004221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721004221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721004222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721004237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721004238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721004339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721004339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721004340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721004352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721004353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,469 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/e5144af0cf214255af351f67282b9ab1 2023-04-05 18:55:44,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721004544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721004545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721004546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721004559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721004559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721004854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721004855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721004855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721004864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:44,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721004866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:44,882 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/dc0c0f3bb9ed40bf9e03b570baf0588f 2023-04-05 18:55:45,293 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/d567c5a660af4049a7cb6c1b1918ad97 2023-04-05 18:55:45,296 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/e5144af0cf214255af351f67282b9ab1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e5144af0cf214255af351f67282b9ab1 2023-04-05 18:55:45,300 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e5144af0cf214255af351f67282b9ab1, entries=150, sequenceid=125, filesize=11.6 K 2023-04-05 18:55:45,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/dc0c0f3bb9ed40bf9e03b570baf0588f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/dc0c0f3bb9ed40bf9e03b570baf0588f 2023-04-05 18:55:45,303 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/dc0c0f3bb9ed40bf9e03b570baf0588f, entries=150, sequenceid=125, filesize=11.6 K 2023-04-05 18:55:45,304 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/d567c5a660af4049a7cb6c1b1918ad97 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d567c5a660af4049a7cb6c1b1918ad97 2023-04-05 18:55:45,307 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d567c5a660af4049a7cb6c1b1918ad97, entries=150, sequenceid=125, filesize=11.6 K 2023-04-05 18:55:45,307 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6e819dddd14a04b6eb4e71fe4e02a010 in 1248ms, sequenceid=125, compaction requested=true 2023-04-05 18:55:45,307 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:45,307 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:45,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:45,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:45,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:45,309 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,309 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ea6f28f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:45,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:45,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,312 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:45,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:45,314 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:45,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:45,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:45,316 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:45,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:45,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:45,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:45,316 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:45,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,321 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,321 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,321 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,321 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:45,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:45,321 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,322 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:45,322 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:45,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:45,368 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 18:55:45,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:45,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:45,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:45,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:45,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:45,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:45,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721005384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721005385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721005390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721005394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721005400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,422 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:45,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:45,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:45,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:45,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:45,425 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:45,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:45,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:45,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:45,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,426 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,426 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,427 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,427 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:45,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:45,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:45,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:45,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:45,428 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:45,430 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e342709[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:45,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:45,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,433 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:45,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:45,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:45,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:45,434 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:45,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:45,436 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:45,436 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:45,436 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:45,436 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:45,436 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:45,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:45,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:45,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,443 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:45,443 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:45,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721005500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721005500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721005501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721005511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721005511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,543 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:45,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:45,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:45,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:45,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:45,547 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:45,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:45,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:45,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:45,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,548 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:45,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:45,550 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,550 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:45,550 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:45,550 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:45,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,552 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,552 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d0a3e3b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:45,553 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:45,553 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,554 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,554 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:45,554 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:45,554 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:45,555 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,555 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:45,557 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:45,564 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:45,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:45,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:45,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:45,564 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:45,566 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,566 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,567 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,567 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,566 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,570 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,570 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,570 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,570 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,570 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,570 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,570 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,570 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,570 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:45,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:45,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:45,571 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:45,571 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:45,671 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:45,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:45,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:45,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:45,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:45,674 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:45,674 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:45,674 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:45,674 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:45,675 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,675 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,676 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,676 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,676 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,676 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:45,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:45,677 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,677 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:45,677 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:45,677 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:45,678 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:45,682 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,682 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c95ddb8[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:45,685 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:45,685 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,686 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:45,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:45,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:45,686 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,686 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:45,687 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,688 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,688 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:45,690 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:45,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:45,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:45,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:45,690 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:45,692 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,692 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,692 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:45,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:45,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,696 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:45,697 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:45,696 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,697 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,697 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,697 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,697 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721005708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721005708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721005709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721005718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:45,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721005718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,775 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/d64e0552d26e4694b4cfdeb4d5242e39 2023-04-05 18:55:45,797 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:45,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:45,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:45,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:45,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:45,800 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:45,800 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:45,800 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:45,800 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:45,801 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,801 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,802 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,802 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,803 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,803 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:45,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:45,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:45,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:45,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:45,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:45,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:45,804 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,804 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:45,804 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:45,804 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,804 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:45,805 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:45,805 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,805 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:45,805 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:45,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152e662f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:45,807 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:45,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,810 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:45,810 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:45,810 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:45,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:45,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:45,813 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:45,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:45,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:45,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:45,813 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:45,816 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,816 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,816 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,817 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,818 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,819 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:45,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:45,820 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:45,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,820 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:45,820 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:45,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,920 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:45,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:45,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:45,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:45,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:45,923 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:45,923 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:45,923 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:45,924 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:45,924 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,924 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,926 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,926 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,927 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,927 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:45,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:45,928 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,928 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:45,928 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:45,928 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:45,929 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:45,929 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,929 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:45,929 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:45,930 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4afd17ec[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:45,932 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:45,932 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,933 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,933 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:45,933 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:45,933 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:45,933 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,933 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:45,934 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:45,936 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:45,936 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:45,936 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:45,936 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:45,936 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:45,938 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:45,940 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,940 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:45,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,941 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:45,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:45,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:45,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:45,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:45,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:45,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:45,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:45,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:45,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:45,943 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:45,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:45,944 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:45,944 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:46,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721006015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721006015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721006016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721006025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721006027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,044 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:46,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:46,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:46,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:46,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:46,047 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:46,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:46,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:46,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:46,048 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,048 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,049 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,049 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:46,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:46,049 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,049 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:46,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:46,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:46,050 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:46,050 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:46,050 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:46,050 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:46,050 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:46,050 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,051 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:46,051 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:46,051 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:46,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:46,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:46,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:46,052 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@662a036e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:46,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:46,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,058 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:46,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:46,062 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:46,062 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:46,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,062 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:46,065 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:46,065 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:46,065 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:46,065 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:46,065 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:46,066 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,066 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,066 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:46,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:46,067 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,068 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,070 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,070 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:46,070 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:46,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:46,071 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:46,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:46,070 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:46,071 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:46,071 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,071 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:46,071 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,071 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,071 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:46,171 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:46,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:46,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:46,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:46,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:46,174 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:46,175 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:46,175 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:46,175 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:46,175 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,175 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,176 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:46,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:46,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:46,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:46,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:46,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:46,180 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,180 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:46,180 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:46,180 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:46,182 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,182 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,182 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,183 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:46,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e06a0e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:46,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:46,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,186 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:46,186 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:46,186 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:46,186 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,186 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:46,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,189 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:46,189 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,189 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:46,191 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:46,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:46,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:46,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:46,191 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:46,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,194 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/2d15eab9b82541138d7206e387aaddcf 2023-04-05 18:55:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,198 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:46,198 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:46,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:46,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:46,198 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:46,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:46,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:46,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:46,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:46,199 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:46,199 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:46,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/bb542525c603428995729d320f72478a 2023-04-05 18:55:46,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/d64e0552d26e4694b4cfdeb4d5242e39 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d64e0552d26e4694b4cfdeb4d5242e39 2023-04-05 18:55:46,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d64e0552d26e4694b4cfdeb4d5242e39, entries=200, sequenceid=144, filesize=14.1 K 2023-04-05 18:55:46,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/2d15eab9b82541138d7206e387aaddcf as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/2d15eab9b82541138d7206e387aaddcf 2023-04-05 18:55:46,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/2d15eab9b82541138d7206e387aaddcf, entries=150, sequenceid=144, filesize=11.8 K 2023-04-05 18:55:46,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/bb542525c603428995729d320f72478a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/bb542525c603428995729d320f72478a 2023-04-05 18:55:46,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/bb542525c603428995729d320f72478a, entries=150, sequenceid=144, filesize=11.8 K 2023-04-05 18:55:46,245 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6e819dddd14a04b6eb4e71fe4e02a010 in 877ms, sequenceid=144, compaction requested=true 2023-04-05 18:55:46,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:46,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:46,245 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 18:55:46,245 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 18:55:46,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:46,246 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:46,249 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97628 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 18:55:46,249 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:55:46,250 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,250 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2c3afd2982114da690d89f1669d2fbdb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/eabeb98b45d9420f89b07c12dcd311aa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/208dbed4bd8747808c5fc3ae9f3414f8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d5ac58e007d34f7aa0343e55f327f2ba, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/56b718db957f49199b9c128414f5ca0f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d9bf50cd79a24b3a9cc69dba8eb28d25, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e5144af0cf214255af351f67282b9ab1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d64e0552d26e4694b4cfdeb4d5242e39] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=95.3 K 2023-04-05 18:55:46,250 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 2c3afd2982114da690d89f1669d2fbdb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680720939299 2023-04-05 18:55:46,250 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95238 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 18:55:46,250 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:55:46,250 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,251 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f62e4bcb945c4336a3a595b1d41ffe2f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/e9700cfe63a847589fe12f1f990a9efa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/983ef6ecd07f40b6901c4eea05f35870, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b712f544f53943baa19796add81929ac, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/57f6a20fa19a400dbf533819520d9bc7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/00c3dd3121b14b5f8ae954d377dc8514, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/dc0c0f3bb9ed40bf9e03b570baf0588f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/2d15eab9b82541138d7206e387aaddcf] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=93.0 K 2023-04-05 18:55:46,251 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting eabeb98b45d9420f89b07c12dcd311aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680720939331 2023-04-05 18:55:46,251 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting f62e4bcb945c4336a3a595b1d41ffe2f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680720939299 2023-04-05 18:55:46,252 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 208dbed4bd8747808c5fc3ae9f3414f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680720939946 2023-04-05 18:55:46,252 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e9700cfe63a847589fe12f1f990a9efa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680720939331 2023-04-05 18:55:46,252 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting d5ac58e007d34f7aa0343e55f327f2ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680720940583 2023-04-05 18:55:46,252 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 56b718db957f49199b9c128414f5ca0f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680720941734 2023-04-05 18:55:46,252 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 983ef6ecd07f40b6901c4eea05f35870, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680720939946 2023-04-05 18:55:46,253 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting d9bf50cd79a24b3a9cc69dba8eb28d25, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680720942925 2023-04-05 18:55:46,253 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting e5144af0cf214255af351f67282b9ab1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680720943573 2023-04-05 18:55:46,255 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting b712f544f53943baa19796add81929ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680720940583 2023-04-05 18:55:46,255 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting d64e0552d26e4694b4cfdeb4d5242e39, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680720944220 2023-04-05 18:55:46,255 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 57f6a20fa19a400dbf533819520d9bc7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680720941734 2023-04-05 18:55:46,257 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 00c3dd3121b14b5f8ae954d377dc8514, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680720942925 2023-04-05 18:55:46,258 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting dc0c0f3bb9ed40bf9e03b570baf0588f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680720943573 2023-04-05 18:55:46,258 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 2d15eab9b82541138d7206e387aaddcf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680720944220 2023-04-05 18:55:46,285 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#413 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:46,287 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#414 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:46,299 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:46,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:46,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:46,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:46,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:46,310 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:46,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:46,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:46,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:46,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,312 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:46,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:46,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,313 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/294c81e8d85240339dc4f675607cabd6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/294c81e8d85240339dc4f675607cabd6 2023-04-05 18:55:46,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:46,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:46,315 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/c915c48c5ac64c1b95ed705904cf7f2a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c915c48c5ac64c1b95ed705904cf7f2a 2023-04-05 18:55:46,318 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into 294c81e8d85240339dc4f675607cabd6(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:46,318 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:46,318 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=8, startTime=1680720946245; duration=0sec 2023-04-05 18:55:46,318 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:46,318 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 18:55:46,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:46,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:46,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:46,319 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 18:55:46,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:46,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:46,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:46,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:46,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:46,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:46,321 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95238 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 18:55:46,321 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:55:46,321 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,321 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4456fe4c1c8449a0b835b8d94b308f16, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0ff12166cf9b4f6a8ac47bf9f0ff8804, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/ab7cdb51e4964425b38d5792f8540da0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/928052bb087a435dafdd44c505dc0e32, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32d69cda5ca945c8a44fde188ebe7cd5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0819993c38ef497abcc238bb9d449fb9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d567c5a660af4049a7cb6c1b1918ad97, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/bb542525c603428995729d320f72478a] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=93.0 K 2023-04-05 18:55:46,321 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4456fe4c1c8449a0b835b8d94b308f16, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680720939299 2023-04-05 18:55:46,322 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0ff12166cf9b4f6a8ac47bf9f0ff8804, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680720939331 2023-04-05 18:55:46,322 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ab7cdb51e4964425b38d5792f8540da0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680720939946 2023-04-05 18:55:46,322 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 928052bb087a435dafdd44c505dc0e32, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680720940583 2023-04-05 18:55:46,323 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 32d69cda5ca945c8a44fde188ebe7cd5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680720941734 2023-04-05 18:55:46,323 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0819993c38ef497abcc238bb9d449fb9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680720942925 2023-04-05 18:55:46,323 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting d567c5a660af4049a7cb6c1b1918ad97, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680720943573 2023-04-05 18:55:46,324 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting bb542525c603428995729d320f72478a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680720944220 2023-04-05 18:55:46,329 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into c915c48c5ac64c1b95ed705904cf7f2a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:46,329 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:46,329 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=8, startTime=1680720946245; duration=0sec 2023-04-05 18:55:46,329 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:46,336 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/b06f330ce0214079b8bc8dae6901f609 2023-04-05 18:55:46,338 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#416 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:46,347 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/1aa9a1a7e297421abbcc87231ea06108 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1aa9a1a7e297421abbcc87231ea06108 2023-04-05 18:55:46,350 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/01c85b5b24d748e59a135d8b2a90aebf 2023-04-05 18:55:46,352 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into 1aa9a1a7e297421abbcc87231ea06108(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:46,353 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:46,353 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=8, startTime=1680720946246; duration=0sec 2023-04-05 18:55:46,353 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:46,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:46,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:46,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721006545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721006545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721006548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721006549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721006551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721006652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721006652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721006652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721006657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721006659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,769 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/8b8d76af55314e9ab2b385b88098704c 2023-04-05 18:55:46,773 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/b06f330ce0214079b8bc8dae6901f609 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b06f330ce0214079b8bc8dae6901f609 2023-04-05 18:55:46,776 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b06f330ce0214079b8bc8dae6901f609, entries=150, sequenceid=158, filesize=11.8 K 2023-04-05 18:55:46,776 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/01c85b5b24d748e59a135d8b2a90aebf as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/01c85b5b24d748e59a135d8b2a90aebf 2023-04-05 18:55:46,779 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/01c85b5b24d748e59a135d8b2a90aebf, entries=150, sequenceid=158, filesize=11.8 K 2023-04-05 18:55:46,780 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/8b8d76af55314e9ab2b385b88098704c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/8b8d76af55314e9ab2b385b88098704c 2023-04-05 18:55:46,783 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/8b8d76af55314e9ab2b385b88098704c, entries=150, sequenceid=158, filesize=11.8 K 2023-04-05 18:55:46,784 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6e819dddd14a04b6eb4e71fe4e02a010 in 465ms, sequenceid=158, compaction requested=false 2023-04-05 18:55:46,784 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:46,784 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:46,786 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321971a3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:46,788 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:46,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,790 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,790 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,791 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:46,791 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:46,791 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:46,791 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:46,792 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:46,793 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,793 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:46,795 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:46,795 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:46,795 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:46,795 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:46,795 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:46,796 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,796 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,797 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,797 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,798 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,801 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:46,801 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,801 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:46,801 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:46,801 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:46,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:46,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:46,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:46,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:46,801 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,802 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:46,802 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,802 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:46,802 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,802 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:46,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:46,802 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:46,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:55:46,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:46,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:46,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:46,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:46,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:46,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:46,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/6307f23be8194600955aaf864f2b4a13 2023-04-05 18:55:46,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721006868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721006869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/76a5dd5795b142269b3457b69cd8910e 2023-04-05 18:55:46,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721006872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721006875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721006876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,902 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:46,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:46,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:46,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:46,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:46,904 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:46,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:46,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:46,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:46,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,906 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:46,906 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:46,907 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,907 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:46,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:46,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:46,908 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,908 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:46,908 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:46,908 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:46,910 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f9427b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:46,911 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:46,911 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,912 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:46,912 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,913 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:46,914 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,915 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:46,915 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:46,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:46,916 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:46,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:46,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:46,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:46,917 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:46,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:46,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:46,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:46,924 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,925 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,927 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:46,927 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:46,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:46,927 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:46,927 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:46,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:46,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:46,928 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,928 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:46,928 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,928 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:46,928 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:46,928 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,928 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:46,928 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:46,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:46,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721006977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721006977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721006979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721006980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:46,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:46,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721006985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,028 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:47,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:47,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:47,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:47,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:47,040 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:47,040 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,040 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:47,040 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:47,040 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,040 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,042 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,042 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,042 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,042 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:47,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:47,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:47,048 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:47,048 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,048 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:47,050 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,050 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:47,051 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:47,051 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:47,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:47,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:47,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:47,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,052 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26fe1279[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:47,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:47,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,057 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:47,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:47,058 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:47,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:47,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:47,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:47,062 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:47,062 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:47,062 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:47,062 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:47,062 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:47,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,063 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,064 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,064 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,066 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:47,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:47,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,070 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,071 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:47,071 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:47,171 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:47,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:47,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:47,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:47,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:47,174 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:47,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:47,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:47,175 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,175 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,176 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,177 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,177 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:47,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:47,178 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,178 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:47,178 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:47,178 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:47,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:47,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:47,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:47,180 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,181 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7199acb2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:47,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721007181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721007182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:47,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,184 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:47,185 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,185 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:47,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,186 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:47,186 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,186 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:47,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:47,188 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:47,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:47,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:47,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:47,189 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:47,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721007184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721007186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721007188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,197 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:47,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:47,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:47,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,199 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:47,199 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:47,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,297 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/3b53d6bfb3194852b995061562a029b1 2023-04-05 18:55:47,299 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:47,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:47,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/6307f23be8194600955aaf864f2b4a13 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6307f23be8194600955aaf864f2b4a13 2023-04-05 18:55:47,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:47,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:47,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:47,303 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:47,303 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,303 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:47,303 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:47,303 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,304 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6307f23be8194600955aaf864f2b4a13, entries=200, sequenceid=180, filesize=14.1 K 2023-04-05 18:55:47,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:47,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/76a5dd5795b142269b3457b69cd8910e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/76a5dd5795b142269b3457b69cd8910e 2023-04-05 18:55:47,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:47,307 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,307 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:47,307 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:47,307 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,307 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:47,307 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:47,307 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,307 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:47,307 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:47,312 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,314 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/76a5dd5795b142269b3457b69cd8910e, entries=150, sequenceid=180, filesize=11.8 K 2023-04-05 18:55:47,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:47,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8f45f4b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:47,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:47,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/3b53d6bfb3194852b995061562a029b1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3b53d6bfb3194852b995061562a029b1 2023-04-05 18:55:47,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3b53d6bfb3194852b995061562a029b1, entries=150, sequenceid=180, filesize=11.8 K 2023-04-05 18:55:47,318 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6e819dddd14a04b6eb4e71fe4e02a010 in 459ms, sequenceid=180, compaction requested=true 2023-04-05 18:55:47,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:47,318 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:47,318 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:47,318 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:47,318 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:47,318 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:47,319 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:47,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:47,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:47,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:47,319 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:47,319 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:55:47,319 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,319 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38772 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:47,319 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/294c81e8d85240339dc4f675607cabd6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/01c85b5b24d748e59a135d8b2a90aebf, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/76a5dd5795b142269b3457b69cd8910e] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=35.5 K 2023-04-05 18:55:47,319 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:55:47,320 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,320 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c915c48c5ac64c1b95ed705904cf7f2a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b06f330ce0214079b8bc8dae6901f609, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6307f23be8194600955aaf864f2b4a13] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=37.9 K 2023-04-05 18:55:47,320 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 294c81e8d85240339dc4f675607cabd6, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680720944220 2023-04-05 18:55:47,320 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,320 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c915c48c5ac64c1b95ed705904cf7f2a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680720944220 2023-04-05 18:55:47,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:47,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,320 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting b06f330ce0214079b8bc8dae6901f609, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680720945384 2023-04-05 18:55:47,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:47,320 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 01c85b5b24d748e59a135d8b2a90aebf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680720945384 2023-04-05 18:55:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,321 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6307f23be8194600955aaf864f2b4a13, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680720946546 2023-04-05 18:55:47,321 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 76a5dd5795b142269b3457b69cd8910e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680720946546 2023-04-05 18:55:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:47,323 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:47,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:47,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:47,323 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:47,323 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:47,325 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,326 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,327 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#422 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:47,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,329 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:47,333 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:47,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:47,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:47,333 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:47,334 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:47,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,347 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/899ec81701d242c292be2d28420d1fa7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/899ec81701d242c292be2d28420d1fa7 2023-04-05 18:55:47,351 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into 899ec81701d242c292be2d28420d1fa7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:47,351 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:47,351 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=13, startTime=1680720947318; duration=0sec 2023-04-05 18:55:47,351 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:47,351 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:47,351 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:47,352 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:55:47,352 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,352 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1aa9a1a7e297421abbcc87231ea06108, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/8b8d76af55314e9ab2b385b88098704c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3b53d6bfb3194852b995061562a029b1] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=35.5 K 2023-04-05 18:55:47,352 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 1aa9a1a7e297421abbcc87231ea06108, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680720944220 2023-04-05 18:55:47,352 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 8b8d76af55314e9ab2b385b88098704c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680720945384 2023-04-05 18:55:47,352 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 3b53d6bfb3194852b995061562a029b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680720946546 2023-04-05 18:55:47,357 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#424 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:47,366 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/d4c71c22b4dd459aa4a0d31b45bde312 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d4c71c22b4dd459aa4a0d31b45bde312 2023-04-05 18:55:47,369 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into d4c71c22b4dd459aa4a0d31b45bde312(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:47,369 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:47,369 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=13, startTime=1680720947318; duration=0sec 2023-04-05 18:55:47,369 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:47,434 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:47,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:47,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:47,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:47,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:47,437 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:47,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:47,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:47,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,438 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:47,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:47,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:47,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:47,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:47,441 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,441 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:47,441 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:55:47,441 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:47,441 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:47,441 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:47,441 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:47,441 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:47,441 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:47,448 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/198683004234478dbeee8b58b25dbe38 2023-04-05 18:55:47,456 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/868c3038ea024ccc9f49fdda5e5b2500 2023-04-05 18:55:47,466 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/32268a80bbb344a7a47c66f7bd475583 2023-04-05 18:55:47,470 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/198683004234478dbeee8b58b25dbe38 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/198683004234478dbeee8b58b25dbe38 2023-04-05 18:55:47,473 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/198683004234478dbeee8b58b25dbe38, entries=150, sequenceid=194, filesize=11.8 K 2023-04-05 18:55:47,473 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/868c3038ea024ccc9f49fdda5e5b2500 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/868c3038ea024ccc9f49fdda5e5b2500 2023-04-05 18:55:47,478 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/868c3038ea024ccc9f49fdda5e5b2500, entries=150, sequenceid=194, filesize=11.8 K 2023-04-05 18:55:47,479 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/32268a80bbb344a7a47c66f7bd475583 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32268a80bbb344a7a47c66f7bd475583 2023-04-05 18:55:47,482 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32268a80bbb344a7a47c66f7bd475583, entries=150, sequenceid=194, filesize=11.8 K 2023-04-05 18:55:47,482 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 6e819dddd14a04b6eb4e71fe4e02a010 in 41ms, sequenceid=194, compaction requested=false 2023-04-05 18:55:47,482 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:47,483 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,484 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:47,484 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:47,485 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,485 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52443eb4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:47,487 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:47,487 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,489 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:47,489 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,489 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,491 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:47,494 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:47,494 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:47,494 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:47,495 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:47,495 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:47,496 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,496 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,496 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,497 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:47,498 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,499 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:47,499 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:47,499 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:47,499 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:47,499 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:47,499 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:47,499 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,507 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,507 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,507 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:47,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:47,508 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:47,508 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:47,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/c6ac4c6027644456805cca292267db08 2023-04-05 18:55:47,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721007522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721007523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721007525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721007525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721007527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/ba6c637b1c8648719c715ad80aeb2f2b 2023-04-05 18:55:47,608 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:47,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:47,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:47,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:47,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:47,611 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:47,611 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,611 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:47,611 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:47,612 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,612 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,613 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,613 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,613 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,613 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:47,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:47,615 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,615 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:47,615 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:47,615 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:47,617 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:47,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47f935f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:47,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:47,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,620 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:47,620 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:47,620 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:47,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:47,621 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:47,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:47,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:47,625 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:47,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:47,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:47,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:47,625 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:47,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,628 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:47,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:47,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,634 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,634 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:47,634 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:47,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721007632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721007632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721007635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721007636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721007637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,734 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:47,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:47,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:47,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:47,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:47,737 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:47,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:47,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:47,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,739 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,739 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,739 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:47,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:47,740 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,740 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:47,741 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:47,741 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:47,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:47,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:47,741 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:47,742 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:47,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@383f435e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:47,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:47,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:47,746 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,746 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:47,746 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/bf5c18568ee846f897f20c3a5459e456 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bf5c18568ee846f897f20c3a5459e456 2023-04-05 18:55:47,749 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,749 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:47,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:47,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,751 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into bf5c18568ee846f897f20c3a5459e456(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:47,751 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:47,751 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=13, startTime=1680720947318; duration=0sec 2023-04-05 18:55:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,751 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:47,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:47,752 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:47,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:47,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:47,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:47,752 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:47,754 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,754 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,754 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,755 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,756 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:47,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:47,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,759 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,759 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:47,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,759 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:47,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:47,759 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,759 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721007837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721007837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721007840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721007840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721007842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,859 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:47,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:47,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:47,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:47,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:47,862 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:47,862 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,862 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:47,862 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:47,862 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,862 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,866 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,866 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,867 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,867 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:47,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:47,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:47,867 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:47,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:47,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:47,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:47,868 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,868 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:47,868 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:47,868 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:47,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:47,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:47,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:47,870 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,870 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,871 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,871 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121b4d15[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:47,872 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:47,872 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,873 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:47,873 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,873 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:47,874 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,874 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:47,874 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,875 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:47,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:47,877 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:47,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:47,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:47,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:47,877 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:47,878 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,878 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,878 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:47,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,880 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,882 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,885 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,885 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:47,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:47,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:47,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:47,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,885 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:47,886 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:47,886 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,886 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,886 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,886 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:47,886 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,886 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:47,886 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,954 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/e9d9935804bd4919ae327d6c3199ba20 2023-04-05 18:55:47,958 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/c6ac4c6027644456805cca292267db08 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c6ac4c6027644456805cca292267db08 2023-04-05 18:55:47,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c6ac4c6027644456805cca292267db08, entries=200, sequenceid=206, filesize=14.1 K 2023-04-05 18:55:47,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/ba6c637b1c8648719c715ad80aeb2f2b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ba6c637b1c8648719c715ad80aeb2f2b 2023-04-05 18:55:47,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ba6c637b1c8648719c715ad80aeb2f2b, entries=150, sequenceid=206, filesize=11.8 K 2023-04-05 18:55:47,965 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/e9d9935804bd4919ae327d6c3199ba20 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e9d9935804bd4919ae327d6c3199ba20 2023-04-05 18:55:47,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e9d9935804bd4919ae327d6c3199ba20, entries=150, sequenceid=206, filesize=11.8 K 2023-04-05 18:55:47,969 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6e819dddd14a04b6eb4e71fe4e02a010 in 471ms, sequenceid=206, compaction requested=true 2023-04-05 18:55:47,969 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:47,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:47,970 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:47,970 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:47,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:47,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:47,970 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38874 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:47,971 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:47,971 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:55:47,971 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:55:47,971 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,971 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,971 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/899ec81701d242c292be2d28420d1fa7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/198683004234478dbeee8b58b25dbe38, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c6ac4c6027644456805cca292267db08] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=38.0 K 2023-04-05 18:55:47,971 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bf5c18568ee846f897f20c3a5459e456, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/868c3038ea024ccc9f49fdda5e5b2500, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ba6c637b1c8648719c715ad80aeb2f2b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=35.6 K 2023-04-05 18:55:47,971 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 899ec81701d242c292be2d28420d1fa7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680720946546 2023-04-05 18:55:47,971 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting bf5c18568ee846f897f20c3a5459e456, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680720946546 2023-04-05 18:55:47,972 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 198683004234478dbeee8b58b25dbe38, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680720946867 2023-04-05 18:55:47,972 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 868c3038ea024ccc9f49fdda5e5b2500, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680720946867 2023-04-05 18:55:47,972 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c6ac4c6027644456805cca292267db08, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680720947489 2023-04-05 18:55:47,972 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ba6c637b1c8648719c715ad80aeb2f2b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680720947495 2023-04-05 18:55:47,979 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#431 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:47,981 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:47,986 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:47,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:47,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:47,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:47,989 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:47,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:47,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:47,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,989 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/50515664a0e1495da01aa7901ae81741 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/50515664a0e1495da01aa7901ae81741 2023-04-05 18:55:47,990 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:47,990 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,991 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:47,991 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:47,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:47,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:47,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:47,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:47,992 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,992 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:47,992 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:55:47,992 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:47,993 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:47,993 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:47,993 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:47,993 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:47,993 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:47,997 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into 50515664a0e1495da01aa7901ae81741(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:47,998 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:47,998 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=13, startTime=1680720947970; duration=0sec 2023-04-05 18:55:47,998 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:47,998 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:47,998 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:47,999 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:55:47,999 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:47,999 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d4c71c22b4dd459aa4a0d31b45bde312, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32268a80bbb344a7a47c66f7bd475583, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e9d9935804bd4919ae327d6c3199ba20] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=35.6 K 2023-04-05 18:55:47,999 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting d4c71c22b4dd459aa4a0d31b45bde312, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680720946546 2023-04-05 18:55:47,999 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 32268a80bbb344a7a47c66f7bd475583, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680720946867 2023-04-05 18:55:48,000 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e9d9935804bd4919ae327d6c3199ba20, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680720947495 2023-04-05 18:55:48,008 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/2d992e7af1f44453a935076c350bb731 2023-04-05 18:55:48,009 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#434 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:48,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:48,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:48,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721008156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721008157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721008161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721008162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721008163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721008263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721008264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721008270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721008271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721008274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,387 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/22c1aa6c50f3411da1920c3ab31eeb4c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/22c1aa6c50f3411da1920c3ab31eeb4c 2023-04-05 18:55:48,391 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into 22c1aa6c50f3411da1920c3ab31eeb4c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:48,391 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:48,391 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=13, startTime=1680720947970; duration=0sec 2023-04-05 18:55:48,392 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:48,416 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/f575b194c80b46ff96d521b01321f4f3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f575b194c80b46ff96d521b01321f4f3 2023-04-05 18:55:48,418 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/ed5070badd7f4e9cb5d41cdf812494b5 2023-04-05 18:55:48,420 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into f575b194c80b46ff96d521b01321f4f3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:48,420 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:48,420 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=13, startTime=1680720947970; duration=0sec 2023-04-05 18:55:48,420 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:48,427 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/f7e7ca277afc4c4bb0440c626487f9df 2023-04-05 18:55:48,431 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/2d992e7af1f44453a935076c350bb731 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2d992e7af1f44453a935076c350bb731 2023-04-05 18:55:48,434 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2d992e7af1f44453a935076c350bb731, entries=150, sequenceid=227, filesize=11.8 K 2023-04-05 18:55:48,434 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/ed5070badd7f4e9cb5d41cdf812494b5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed5070badd7f4e9cb5d41cdf812494b5 2023-04-05 18:55:48,437 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed5070badd7f4e9cb5d41cdf812494b5, entries=150, sequenceid=227, filesize=11.8 K 2023-04-05 18:55:48,438 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/f7e7ca277afc4c4bb0440c626487f9df as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f7e7ca277afc4c4bb0440c626487f9df 2023-04-05 18:55:48,441 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f7e7ca277afc4c4bb0440c626487f9df, entries=150, sequenceid=227, filesize=11.8 K 2023-04-05 18:55:48,441 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 6e819dddd14a04b6eb4e71fe4e02a010 in 449ms, sequenceid=227, compaction requested=false 2023-04-05 18:55:48,441 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:48,441 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:48,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:48,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:48,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:48,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:48,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6159607c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:48,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:48,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,446 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,446 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:48,446 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:48,446 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:48,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:48,451 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,451 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,451 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:48,451 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:48,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:48,453 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:48,453 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:48,453 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:48,453 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:48,453 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:48,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:48,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,455 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:48,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:48,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:48,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:48,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:48,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:48,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:48,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,460 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:48,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,461 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:48,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,461 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:48,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:48,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 18:55:48,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:48,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:48,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:48,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:48,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:48,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:48,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721008489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721008493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721008494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721008495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721008496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,561 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:48,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:48,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:48,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:48,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:48,564 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:48,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:48,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:48,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:48,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,565 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:48,565 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:48,566 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,566 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:48,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:48,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:48,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:48,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:48,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:48,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:48,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:48,567 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,567 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:48,567 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:48,567 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:48,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:48,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:48,567 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:48,569 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,569 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:48,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c4f263[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:48,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:48,571 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,572 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:48,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:48,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:48,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:48,573 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,573 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:48,573 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:48,573 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:48,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:48,575 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:48,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:48,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:48,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:48,575 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:48,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,577 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:48,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:48,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:48,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:48,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:48,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:48,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:48,585 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:48,586 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:48,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721008597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721008597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721008597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721008602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721008602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,686 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:48,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:48,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:48,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:48,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:48,688 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:48,689 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:48,689 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:48,689 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:48,689 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,689 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,690 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:48,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:48,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:48,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:48,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:48,692 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,692 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:48,692 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:48,692 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:48,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:48,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,696 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b2377b6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:48,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:48,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,701 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:48,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,701 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:48,701 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:48,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:48,702 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:48,702 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:48,702 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,702 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:48,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:48,704 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:48,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:48,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:48,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:48,705 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:48,706 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,706 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,706 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:48,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:48,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,709 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,709 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,709 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,710 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:48,709 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:48,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:48,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:48,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:48,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:48,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:48,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:48,710 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:48,710 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:48,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:48,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42020 deadline: 1680721008800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42032 deadline: 1680721008800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42008 deadline: 1680721008801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721008806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,810 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:48,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:48,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:48,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721008806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:48,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:48,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:48,814 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:48,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:48,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:48,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:48,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,816 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:48,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:48,816 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,817 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,817 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:48,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:48,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:48,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:48,818 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,818 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:48,818 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:48,818 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:48,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:48,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:48,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1398ea22[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:48,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:48,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,823 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:48,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:48,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:48,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:48,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:48,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:48,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:48,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:48,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:48,828 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:48,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:48,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:48,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:48,828 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:48,831 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,831 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,831 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:48,831 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,833 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:48,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:48,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:48,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,835 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:48,835 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:48,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:48,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:48,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:48,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:48,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:48,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/a80ce5d4cf8745c39639cf733ce815c8 2023-04-05 18:55:48,896 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/6544d0975dda45a7870276b428ad1edd 2023-04-05 18:55:48,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/43d71b759c2d46a59a6d56341512d310 2023-04-05 18:55:48,936 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:48,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:48,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:48,938 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/a80ce5d4cf8745c39639cf733ce815c8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80ce5d4cf8745c39639cf733ce815c8 2023-04-05 18:55:48,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:48,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:48,940 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:48,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:48,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:48,940 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:48,941 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,941 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,942 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:48,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:48,944 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,946 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,946 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80ce5d4cf8745c39639cf733ce815c8, entries=250, sequenceid=245, filesize=16.4 K 2023-04-05 18:55:48,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:48,947 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:48,947 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:48,947 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:48,947 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:48,947 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:48,947 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,947 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:48,947 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:48,947 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/6544d0975dda45a7870276b428ad1edd as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/6544d0975dda45a7870276b428ad1edd 2023-04-05 18:55:48,947 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:48,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/6544d0975dda45a7870276b428ad1edd, entries=150, sequenceid=245, filesize=11.8 K 2023-04-05 18:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39e822ad[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:48,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:48,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/43d71b759c2d46a59a6d56341512d310 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/43d71b759c2d46a59a6d56341512d310 2023-04-05 18:55:48,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:48,955 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,955 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:48,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:48,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:48,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:48,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,957 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/43d71b759c2d46a59a6d56341512d310, entries=150, sequenceid=245, filesize=11.8 K 2023-04-05 18:55:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,958 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 6e819dddd14a04b6eb4e71fe4e02a010 in 485ms, sequenceid=245, compaction requested=true 2023-04-05 18:55:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,958 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:48,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:48,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:48,958 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:48,958 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,959 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:48,959 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:55:48,959 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,959 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/50515664a0e1495da01aa7901ae81741, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed5070badd7f4e9cb5d41cdf812494b5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/6544d0975dda45a7870276b428ad1edd] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=35.7 K 2023-04-05 18:55:48,959 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:48,959 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:55:48,959 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,960 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 50515664a0e1495da01aa7901ae81741, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680720947495 2023-04-05 18:55:48,960 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/22c1aa6c50f3411da1920c3ab31eeb4c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2d992e7af1f44453a935076c350bb731, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80ce5d4cf8745c39639cf733ce815c8] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=40.4 K 2023-04-05 18:55:48,960 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ed5070badd7f4e9cb5d41cdf812494b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680720947522 2023-04-05 18:55:48,960 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 22c1aa6c50f3411da1920c3ab31eeb4c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680720947495 2023-04-05 18:55:48,960 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 6544d0975dda45a7870276b428ad1edd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680720948162 2023-04-05 18:55:48,960 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 2d992e7af1f44453a935076c350bb731, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680720947522 2023-04-05 18:55:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:48,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:48,961 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:48,961 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting a80ce5d4cf8745c39639cf733ce815c8, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680720948160 2023-04-05 18:55:48,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:48,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:48,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:48,961 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:48,962 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:48,962 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,963 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:48,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:48,963 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:48,967 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#440 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:48,967 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,969 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:48,969 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:48,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,970 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:48,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:48,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:48,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:48,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:48,970 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:48,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:48,970 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:48,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:48,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:48,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:48,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:48,979 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/9c4c40342e3b4d579b745d6e1509cc11 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/9c4c40342e3b4d579b745d6e1509cc11 2023-04-05 18:55:48,982 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into 9c4c40342e3b4d579b745d6e1509cc11(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:48,982 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:48,982 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=13, startTime=1680720948958; duration=0sec 2023-04-05 18:55:48,982 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:48,982 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:48,983 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:48,983 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:55:48,983 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:48,983 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f575b194c80b46ff96d521b01321f4f3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f7e7ca277afc4c4bb0440c626487f9df, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/43d71b759c2d46a59a6d56341512d310] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=35.7 K 2023-04-05 18:55:48,984 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting f575b194c80b46ff96d521b01321f4f3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680720947495 2023-04-05 18:55:48,984 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting f7e7ca277afc4c4bb0440c626487f9df, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680720947522 2023-04-05 18:55:48,984 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 43d71b759c2d46a59a6d56341512d310, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680720948162 2023-04-05 18:55:48,989 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:49,070 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:49,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:49,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:49,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:49,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:49,073 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:49,073 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:49,073 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:49,073 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:49,074 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:49,074 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:49,076 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:49,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:49,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:49,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:49,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:49,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:49,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:49,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:49,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:49,078 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-05 18:55:49,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:49,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:49,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:49,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:49,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:49,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:49,085 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/43253dee81274475b2f6022821976793 2023-04-05 18:55:49,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:49,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:49,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721009121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721009121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1680721009122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721009123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721009124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721009233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721009233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721009233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721009233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721009233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,384 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/1217370e976a4a74b7fa3e64d51c10a7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/1217370e976a4a74b7fa3e64d51c10a7 2023-04-05 18:55:49,391 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into 1217370e976a4a74b7fa3e64d51c10a7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:49,391 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:49,391 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=13, startTime=1680720948958; duration=0sec 2023-04-05 18:55:49,391 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:49,404 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/7583b797fb504076a8dd2c50c8b81d09 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/7583b797fb504076a8dd2c50c8b81d09 2023-04-05 18:55:49,411 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into 7583b797fb504076a8dd2c50c8b81d09(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:49,411 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:49,411 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=13, startTime=1680720948958; duration=0sec 2023-04-05 18:55:49,411 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:49,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721009440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42080 deadline: 1680721009440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721009443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721009443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721009448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,494 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/001ff8594dae47cf9a0fe22086491681 2023-04-05 18:55:49,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721009746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721009746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721009749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721009750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:49,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721009755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,914 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/d8de6755e9ee4506b670e5b03f0020d6 2023-04-05 18:55:49,920 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/43253dee81274475b2f6022821976793 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/43253dee81274475b2f6022821976793 2023-04-05 18:55:49,923 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/43253dee81274475b2f6022821976793, entries=150, sequenceid=263, filesize=11.9 K 2023-04-05 18:55:49,923 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/001ff8594dae47cf9a0fe22086491681 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/001ff8594dae47cf9a0fe22086491681 2023-04-05 18:55:49,926 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/001ff8594dae47cf9a0fe22086491681, entries=150, sequenceid=263, filesize=11.9 K 2023-04-05 18:55:49,927 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/d8de6755e9ee4506b670e5b03f0020d6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d8de6755e9ee4506b670e5b03f0020d6 2023-04-05 18:55:49,931 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d8de6755e9ee4506b670e5b03f0020d6, entries=150, sequenceid=263, filesize=11.9 K 2023-04-05 18:55:49,931 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 6e819dddd14a04b6eb4e71fe4e02a010 in 853ms, sequenceid=263, compaction requested=false 2023-04-05 18:55:49,932 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:49,932 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:49,933 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,933 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:49,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:49,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:49,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:49,934 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7018819[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:49,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:49,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:49,936 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:49,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:49,936 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,936 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:49,942 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:49,945 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:49,945 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:49,945 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:49,945 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:49,945 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:49,946 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:49,946 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:49,946 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:49,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:49,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:49,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:49,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:49,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:49,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:49,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:49,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:49,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:49,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:49,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:49,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:49,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:49,950 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:49,950 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:49,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:49,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:49,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:49,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:50,050 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:50,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:50,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:50,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:50,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:50,054 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:50,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:50,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:50,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:50,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,054 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,055 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:50,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:50,056 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,056 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:50,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:50,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:50,057 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:50,057 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:50,057 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:50,057 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:50,057 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:50,057 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:50,057 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:50,057 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 18:55:50,058 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:50,058 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:50,058 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:50,058 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:50,058 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:50,058 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:50,066 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/5c1ba6bf7d014d2393949c62978443e4 2023-04-05 18:55:50,076 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/d61cf939fc5f4653bc089bddad76875c 2023-04-05 18:55:50,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:50,256 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:50,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721010266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721010268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721010269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721010272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721010272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721010373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721010373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721010375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721010375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721010375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,484 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/18d052da8f8b4c9cb02358e478e7dc9a 2023-04-05 18:55:50,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/5c1ba6bf7d014d2393949c62978443e4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/5c1ba6bf7d014d2393949c62978443e4 2023-04-05 18:55:50,491 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/5c1ba6bf7d014d2393949c62978443e4, entries=150, sequenceid=280, filesize=11.9 K 2023-04-05 18:55:50,491 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/d61cf939fc5f4653bc089bddad76875c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/d61cf939fc5f4653bc089bddad76875c 2023-04-05 18:55:50,495 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/d61cf939fc5f4653bc089bddad76875c, entries=150, sequenceid=280, filesize=11.9 K 2023-04-05 18:55:50,495 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/18d052da8f8b4c9cb02358e478e7dc9a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/18d052da8f8b4c9cb02358e478e7dc9a 2023-04-05 18:55:50,498 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/18d052da8f8b4c9cb02358e478e7dc9a, entries=150, sequenceid=280, filesize=11.9 K 2023-04-05 18:55:50,499 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 6e819dddd14a04b6eb4e71fe4e02a010 in 442ms, sequenceid=280, compaction requested=true 2023-04-05 18:55:50,499 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:50,499 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,501 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47dfd188[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:50,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:50,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,504 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:50,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:50,505 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:50,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:50,508 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:50,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:50,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:50,508 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:50,508 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:50,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,511 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:50,511 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,512 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:50,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:50,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:50,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:50,518 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,518 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,518 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,518 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:50,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:50,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:50,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:50,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,519 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,519 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:50,519 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:50,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:50,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 18:55:50,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:50,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:50,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:50,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:50,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:50,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:50,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721010591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721010592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721010592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721010595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721010596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,619 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:50,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:50,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:50,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:50,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:50,622 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:50,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:50,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:50,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:50,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,623 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:50,623 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:50,624 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,624 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:50,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:50,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:50,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:50,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:50,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:50,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:50,625 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:50,625 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:50,626 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:50,626 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:50,627 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:50,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1342eaab[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:50,629 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:50,629 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,630 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,630 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,630 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:50,632 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:50,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,632 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,632 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:50,636 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:50,636 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:50,636 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:50,636 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:50,636 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:50,637 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,637 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,637 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:50,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:50,638 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,640 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:50,643 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:50,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:50,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:50,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:50,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:50,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:50,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:50,644 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:50,644 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:50,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,644 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721010699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721010699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721010699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721010700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721010700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,744 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:50,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:50,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:50,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:50,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:50,747 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:50,747 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:50,747 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:50,747 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:50,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,749 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:50,749 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:50,749 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,749 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:50,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:50,751 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:50,751 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:50,751 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:50,751 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:50,753 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,754 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,754 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:50,754 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b475f24[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:50,755 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:50,756 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,756 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:50,757 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,757 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:50,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,758 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:50,758 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:50,758 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:50,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:50,761 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:50,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:50,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:50,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:50,761 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:50,766 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,766 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,766 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,767 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,771 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,773 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:50,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:50,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:50,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:50,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:50,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:50,773 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:50,773 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:50,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:50,773 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,774 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,774 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:50,874 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:50,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:50,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:50,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:50,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:50,876 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:50,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:50,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:50,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:50,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,877 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,878 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,878 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:50,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:50,878 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,878 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:50,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:50,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:50,879 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:50,880 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:50,880 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:50,880 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:50,880 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:50,880 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:50,880 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,880 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:50,880 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:50,881 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,881 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,881 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,881 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:50,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@600d4315[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:50,883 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:50,883 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,884 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,885 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:50,885 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:50,885 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:50,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:50,886 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,886 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:50,886 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,886 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:50,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:50,888 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:50,888 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:50,888 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:50,888 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:50,888 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:50,890 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,890 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:50,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,890 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:50,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:50,890 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,893 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:50,893 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:50,893 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:50,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:50,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:50,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:50,894 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:50,894 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:50,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:50,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:50,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:50,895 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721010903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721010903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721010904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721010905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:50,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721010906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:50,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/6b4beb6436924d068fc3622b8844cb57 2023-04-05 18:55:50,994 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:50,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:50,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:50,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:50,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:50,998 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/ed29b398334e4d1984e52057ad724c0a 2023-04-05 18:55:50,998 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:50,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:50,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:50,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:50,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:50,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,001 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:51,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:51,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:51,003 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:51,004 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:51,004 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:51,004 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:51,004 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:51,004 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,004 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:51,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,005 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:51,006 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,006 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,007 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/51d06c4d4474488686b3068a9930ae45 2023-04-05 18:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e79d0d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:51,007 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:51,007 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,010 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:51,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/6b4beb6436924d068fc3622b8844cb57 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6b4beb6436924d068fc3622b8844cb57 2023-04-05 18:55:51,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,011 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:51,012 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:51,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6b4beb6436924d068fc3622b8844cb57, entries=250, sequenceid=299, filesize=16.7 K 2023-04-05 18:55:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/ed29b398334e4d1984e52057ad724c0a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed29b398334e4d1984e52057ad724c0a 2023-04-05 18:55:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:51,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:51,015 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:51,015 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:51,015 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:51,015 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:51,015 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:51,018 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,018 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,018 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:51,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed29b398334e4d1984e52057ad724c0a, entries=150, sequenceid=299, filesize=11.9 K 2023-04-05 18:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:51,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:51,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/51d06c4d4474488686b3068a9930ae45 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/51d06c4d4474488686b3068a9930ae45 2023-04-05 18:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:51,019 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,021 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/51d06c4d4474488686b3068a9930ae45, entries=150, sequenceid=299, filesize=11.9 K 2023-04-05 18:55:51,023 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6e819dddd14a04b6eb4e71fe4e02a010 in 443ms, sequenceid=299, compaction requested=true 2023-04-05 18:55:51,023 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:51,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:51,023 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:51,023 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:51,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:51,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:51,024 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54054 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:51,025 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49174 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:51,025 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:55:51,025 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:55:51,025 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:51,025 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:51,025 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/1217370e976a4a74b7fa3e64d51c10a7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/43253dee81274475b2f6022821976793, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/5c1ba6bf7d014d2393949c62978443e4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6b4beb6436924d068fc3622b8844cb57] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=52.8 K 2023-04-05 18:55:51,025 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/9c4c40342e3b4d579b745d6e1509cc11, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/001ff8594dae47cf9a0fe22086491681, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/d61cf939fc5f4653bc089bddad76875c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed29b398334e4d1984e52057ad724c0a] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=48.0 K 2023-04-05 18:55:51,025 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 1217370e976a4a74b7fa3e64d51c10a7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680720948162 2023-04-05 18:55:51,025 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 9c4c40342e3b4d579b745d6e1509cc11, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680720948162 2023-04-05 18:55:51,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:51,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:51,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:51,026 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 001ff8594dae47cf9a0fe22086491681, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680720948488 2023-04-05 18:55:51,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:51,026 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:51,026 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 43253dee81274475b2f6022821976793, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680720948488 2023-04-05 18:55:51,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:51,026 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:51,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,026 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:51,026 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting d61cf939fc5f4653bc089bddad76875c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680720949119 2023-04-05 18:55:51,026 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 5c1ba6bf7d014d2393949c62978443e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680720949119 2023-04-05 18:55:51,026 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,026 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:51,027 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ed29b398334e4d1984e52057ad724c0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680720950264 2023-04-05 18:55:51,027 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6b4beb6436924d068fc3622b8844cb57, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680720950264 2023-04-05 18:55:51,041 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#452 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:51,045 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#453 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:51,054 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/646ec1b855c042c1b1c70b7535e96224 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/646ec1b855c042c1b1c70b7535e96224 2023-04-05 18:55:51,057 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/22f69b9ccc8942c8ae0cd6beccc99c18 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/22f69b9ccc8942c8ae0cd6beccc99c18 2023-04-05 18:55:51,059 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into 646ec1b855c042c1b1c70b7535e96224(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:51,059 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:51,060 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=12, startTime=1680720951023; duration=0sec 2023-04-05 18:55:51,060 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:51,060 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:51,062 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into 22f69b9ccc8942c8ae0cd6beccc99c18(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:51,062 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:51,062 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=12, startTime=1680720951023; duration=0sec 2023-04-05 18:55:51,062 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:51,063 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49174 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:51,063 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:55:51,063 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:51,064 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/7583b797fb504076a8dd2c50c8b81d09, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d8de6755e9ee4506b670e5b03f0020d6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/18d052da8f8b4c9cb02358e478e7dc9a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/51d06c4d4474488686b3068a9930ae45] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=48.0 K 2023-04-05 18:55:51,064 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 7583b797fb504076a8dd2c50c8b81d09, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680720948162 2023-04-05 18:55:51,064 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting d8de6755e9ee4506b670e5b03f0020d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680720948488 2023-04-05 18:55:51,065 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 18d052da8f8b4c9cb02358e478e7dc9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680720949119 2023-04-05 18:55:51,065 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 51d06c4d4474488686b3068a9930ae45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680720950264 2023-04-05 18:55:51,070 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#454 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:51,076 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/14af12d18df149089453fc6798e0b1e4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/14af12d18df149089453fc6798e0b1e4 2023-04-05 18:55:51,080 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into 14af12d18df149089453fc6798e0b1e4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:51,080 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:51,080 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=12, startTime=1680720951023; duration=0sec 2023-04-05 18:55:51,080 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:51,126 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:51,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:51,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:51,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:51,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:51,129 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:51,129 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:51,129 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:51,129 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:51,130 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,130 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,131 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:51,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:51,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:51,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:51,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:51,132 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:51,132 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:51,133 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 18:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:51,143 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/3666baf3790f40b1abb224729cf13240 2023-04-05 18:55:51,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:51,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:51,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721011230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721011234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721011234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721011236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721011236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721011338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721011341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721011341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721011341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721011341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721011540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,552 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/04cfed0a136e4ac1b7a698e29cb54500 2023-04-05 18:55:51,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721011549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721011549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721011550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721011550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,572 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/07401a3d82fb47d583d5ec36ecf815a1 2023-04-05 18:55:51,575 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/3666baf3790f40b1abb224729cf13240 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/3666baf3790f40b1abb224729cf13240 2023-04-05 18:55:51,579 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/3666baf3790f40b1abb224729cf13240, entries=150, sequenceid=315, filesize=11.9 K 2023-04-05 18:55:51,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/04cfed0a136e4ac1b7a698e29cb54500 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/04cfed0a136e4ac1b7a698e29cb54500 2023-04-05 18:55:51,583 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/04cfed0a136e4ac1b7a698e29cb54500, entries=150, sequenceid=315, filesize=11.9 K 2023-04-05 18:55:51,583 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/07401a3d82fb47d583d5ec36ecf815a1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/07401a3d82fb47d583d5ec36ecf815a1 2023-04-05 18:55:51,586 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/07401a3d82fb47d583d5ec36ecf815a1, entries=150, sequenceid=315, filesize=11.9 K 2023-04-05 18:55:51,587 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6e819dddd14a04b6eb4e71fe4e02a010 in 454ms, sequenceid=315, compaction requested=false 2023-04-05 18:55:51,587 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:51,587 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:51,588 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:51,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,589 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:51,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:51,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5df43328[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:51,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:51,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,592 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,592 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:51,592 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:51,592 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:51,592 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,592 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:51,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:51,593 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:51,593 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:51,593 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:51,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:51,595 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:51,595 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:51,595 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:51,595 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:51,595 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:51,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:51,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,600 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:51,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,604 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:51,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:51,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:51,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:51,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:51,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:51,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,604 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:51,605 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:51,605 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:51,605 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,705 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:51,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:51,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:51,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:51,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:51,708 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:51,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:51,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:51,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:51,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,709 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:51,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:51,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,710 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:51,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:51,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:51,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:51,711 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 18:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:51,720 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/248c3c7fb452477cae0aabe95e6c5fb2 2023-04-05 18:55:51,731 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/727bde48b9fd4a60b4efeed7160fa0f6 2023-04-05 18:55:51,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:51,860 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:51,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721011863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721011868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721011872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721011872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721011873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721011974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721011974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721011974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721011974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:51,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:51,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721011979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,144 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/eb4af94ef8594ded98a78ab4552f5bb8 2023-04-05 18:55:52,149 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/248c3c7fb452477cae0aabe95e6c5fb2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/248c3c7fb452477cae0aabe95e6c5fb2 2023-04-05 18:55:52,154 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/248c3c7fb452477cae0aabe95e6c5fb2, entries=150, sequenceid=334, filesize=11.9 K 2023-04-05 18:55:52,155 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/727bde48b9fd4a60b4efeed7160fa0f6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/727bde48b9fd4a60b4efeed7160fa0f6 2023-04-05 18:55:52,158 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/727bde48b9fd4a60b4efeed7160fa0f6, entries=150, sequenceid=334, filesize=11.9 K 2023-04-05 18:55:52,158 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/eb4af94ef8594ded98a78ab4552f5bb8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/eb4af94ef8594ded98a78ab4552f5bb8 2023-04-05 18:55:52,164 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/eb4af94ef8594ded98a78ab4552f5bb8, entries=150, sequenceid=334, filesize=11.9 K 2023-04-05 18:55:52,165 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6e819dddd14a04b6eb4e71fe4e02a010 in 454ms, sequenceid=334, compaction requested=true 2023-04-05 18:55:52,165 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:52,165 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:52,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:52,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:52,165 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:52,168 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,169 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,169 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,169 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:52,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9bf9cd[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:52,170 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:52,170 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,171 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,171 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:52,171 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:52,171 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:52,171 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,171 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:52,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:52,173 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,173 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:52,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:52,176 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:52,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:52,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:52,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:52,176 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:52,178 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,178 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,178 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,185 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:52,187 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 18:55:52,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:52,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:52,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:52,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:52,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:52,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:52,192 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:52,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:52,193 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,193 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:52,193 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:52,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/9d9dbe5617084600acfd51d31034b234 2023-04-05 18:55:52,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721012211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721012212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721012213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721012213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721012214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,293 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:52,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:52,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:52,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:52,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:52,296 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:52,296 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,296 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:52,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:52,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,297 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,298 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,298 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,298 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,298 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:52,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:52,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:52,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:52,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,299 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:52,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:52,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:52,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:52,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:52,300 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:52,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:52,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:52,300 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:52,301 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,301 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60416a0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:52,304 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:52,304 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:52,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,305 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:52,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:52,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:52,307 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:52,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:52,311 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:52,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:52,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:52,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:52,311 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:52,312 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,312 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,312 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721012314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:52,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:52,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,318 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:52,318 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:52,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:52,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721012321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721012336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721012338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721012338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,418 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:52,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:52,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:52,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:52,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:52,421 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:52,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:52,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:52,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,422 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,423 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,423 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:52,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:52,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:52,424 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,424 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:52,424 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:52,424 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:52,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:52,427 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,427 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,427 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,427 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:52,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@636eec98[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:52,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:52,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,430 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:52,430 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,430 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:52,431 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,431 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:52,431 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,431 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:52,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:52,433 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:52,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:52,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:52,434 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:52,434 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:52,435 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,435 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,435 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,435 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:52,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:52,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:52,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,442 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,442 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:52,442 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:52,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,443 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721012518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721012530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,542 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:52,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:52,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:52,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:52,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:52,545 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:52,545 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,545 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:52,545 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:52,546 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,546 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,547 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721012543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721012544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721012544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:52,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:52,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:52,549 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,549 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:52,549 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:52,549 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:52,550 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:52,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:52,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ffffeae[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:52,552 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:52,552 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,553 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:52,554 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:52,554 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:52,554 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,554 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:52,556 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:52,556 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,556 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,556 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:52,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:52,558 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:52,558 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:52,558 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:52,558 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:52,558 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:52,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,560 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,560 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,561 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,563 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,563 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,563 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:52,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:52,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,564 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:52,564 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:52,563 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,564 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/b8cd06877a18472b8789894cb3764ff6 2023-04-05 18:55:52,664 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:52,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:52,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:52,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:52,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:52,667 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:52,667 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,667 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:52,667 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:52,668 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,668 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:52,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:52,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:52,671 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,671 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:52,671 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:52,671 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,674 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:52,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cf9290d[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:52,676 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:52,676 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,677 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:52,677 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,677 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,678 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:52,681 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:52,681 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:52,681 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:52,681 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:52,682 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:52,683 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,683 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,683 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,685 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,688 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,688 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,689 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,689 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,689 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:52,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:52,689 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,689 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,689 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,689 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,689 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,689 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:52,689 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:52,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:52,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,789 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:52,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:52,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:52,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:52,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:52,794 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:52,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:52,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:52,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,795 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,795 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,796 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:52,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:52,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:52,797 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,797 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:52,797 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:52,797 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:52,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:52,798 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,798 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:52,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6e9724[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:52,800 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:52,800 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,801 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:52,802 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,802 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:52,803 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:52,803 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,803 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,803 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:52,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:52,806 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:52,806 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:52,806 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:52,806 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:52,806 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:52,807 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,807 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,807 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,809 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,811 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:52,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:52,811 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,812 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,812 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:52,812 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:52,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:52,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721012825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721012836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721012848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721012849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:52,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721012851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,912 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:52,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:52,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:52,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:52,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:52,915 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:52,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:52,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:52,916 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,916 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,917 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:52,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:52,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:52,919 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:52,919 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:52,919 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,919 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:52,919 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:52,919 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:52,919 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:52,919 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:52,919 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,919 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:52,919 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:52,925 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,925 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,925 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,925 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:52,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e70acdd[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:52,927 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:52,927 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,930 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:52,930 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:52,932 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:52,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:52,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:52,935 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:52,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:52,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:52,935 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:52,935 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:52,937 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,937 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:52,937 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,938 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:52,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,941 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:52,941 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:52,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:52,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:52,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:52,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:52,941 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,941 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:52,942 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:52,942 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:52,942 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:52,942 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:52,942 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:52,942 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,942 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:52,942 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/3e93789a210a44ceb7962f38487f70d2 2023-04-05 18:55:53,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/9d9dbe5617084600acfd51d31034b234 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/9d9dbe5617084600acfd51d31034b234 2023-04-05 18:55:53,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/9d9dbe5617084600acfd51d31034b234, entries=200, sequenceid=348, filesize=14.3 K 2023-04-05 18:55:53,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/b8cd06877a18472b8789894cb3764ff6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b8cd06877a18472b8789894cb3764ff6 2023-04-05 18:55:53,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b8cd06877a18472b8789894cb3764ff6, entries=150, sequenceid=348, filesize=11.9 K 2023-04-05 18:55:53,031 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/3e93789a210a44ceb7962f38487f70d2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3e93789a210a44ceb7962f38487f70d2 2023-04-05 18:55:53,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3e93789a210a44ceb7962f38487f70d2, entries=150, sequenceid=348, filesize=11.9 K 2023-04-05 18:55:53,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6e819dddd14a04b6eb4e71fe4e02a010 in 848ms, sequenceid=348, compaction requested=true 2023-04-05 18:55:53,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:53,035 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:53,035 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:53,035 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:53,035 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:53,035 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:53,036 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:53,036 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:55:53,036 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:53,036 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,036 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:55:53,037 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/22f69b9ccc8942c8ae0cd6beccc99c18, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/3666baf3790f40b1abb224729cf13240, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/248c3c7fb452477cae0aabe95e6c5fb2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/9d9dbe5617084600acfd51d31034b234] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=50.7 K 2023-04-05 18:55:53,037 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,037 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/646ec1b855c042c1b1c70b7535e96224, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/04cfed0a136e4ac1b7a698e29cb54500, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/727bde48b9fd4a60b4efeed7160fa0f6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b8cd06877a18472b8789894cb3764ff6] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=48.3 K 2023-04-05 18:55:53,037 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 22f69b9ccc8942c8ae0cd6beccc99c18, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680720950264 2023-04-05 18:55:53,037 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 646ec1b855c042c1b1c70b7535e96224, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680720950264 2023-04-05 18:55:53,037 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 3666baf3790f40b1abb224729cf13240, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680720950590 2023-04-05 18:55:53,037 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 04cfed0a136e4ac1b7a698e29cb54500, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680720950590 2023-04-05 18:55:53,037 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 248c3c7fb452477cae0aabe95e6c5fb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680720951233 2023-04-05 18:55:53,038 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 727bde48b9fd4a60b4efeed7160fa0f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680720951233 2023-04-05 18:55:53,038 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 9d9dbe5617084600acfd51d31034b234, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680720951867 2023-04-05 18:55:53,038 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting b8cd06877a18472b8789894cb3764ff6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680720951867 2023-04-05 18:55:53,042 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:53,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:53,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:53,046 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:53,046 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,046 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:53,046 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:53,046 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#464 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:53,047 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#465 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:53,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,048 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,049 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,049 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,049 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,049 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:53,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:53,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:53,056 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,056 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:53,056 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 18:55:53,057 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:53,057 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:53,057 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:53,057 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:53,057 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:53,057 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:53,062 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/07fb969d0072401997e9c6e976ba2d3b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/07fb969d0072401997e9c6e976ba2d3b 2023-04-05 18:55:53,064 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/85a61d3baa174f8a9ce337a7ee4400d8 2023-04-05 18:55:53,064 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/de5b57497fbd49ed9c781e775caa9165 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/de5b57497fbd49ed9c781e775caa9165 2023-04-05 18:55:53,067 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into 07fb969d0072401997e9c6e976ba2d3b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:53,067 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:53,067 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=12, startTime=1680720953035; duration=0sec 2023-04-05 18:55:53,067 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:53,067 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:53,069 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:53,069 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:55:53,069 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,069 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/14af12d18df149089453fc6798e0b1e4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/07401a3d82fb47d583d5ec36ecf815a1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/eb4af94ef8594ded98a78ab4552f5bb8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3e93789a210a44ceb7962f38487f70d2] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=48.3 K 2023-04-05 18:55:53,070 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 14af12d18df149089453fc6798e0b1e4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680720950264 2023-04-05 18:55:53,070 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 07401a3d82fb47d583d5ec36ecf815a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680720950590 2023-04-05 18:55:53,071 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into de5b57497fbd49ed9c781e775caa9165(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:53,071 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:53,071 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=12, startTime=1680720953035; duration=0sec 2023-04-05 18:55:53,071 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:53,071 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting eb4af94ef8594ded98a78ab4552f5bb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680720951233 2023-04-05 18:55:53,071 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 3e93789a210a44ceb7962f38487f70d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680720951867 2023-04-05 18:55:53,081 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:53,082 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/1bcc2d521fb048a5b1378074a08e200a 2023-04-05 18:55:53,093 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/93b0df5adc734e82b84ad399c052b390 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/93b0df5adc734e82b84ad399c052b390 2023-04-05 18:55:53,096 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into 93b0df5adc734e82b84ad399c052b390(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:53,096 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:53,096 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=12, startTime=1680720953035; duration=0sec 2023-04-05 18:55:53,096 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:53,102 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/67760f112f0640978b07f36b5a9c0ab7 2023-04-05 18:55:53,106 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/85a61d3baa174f8a9ce337a7ee4400d8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/85a61d3baa174f8a9ce337a7ee4400d8 2023-04-05 18:55:53,109 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/85a61d3baa174f8a9ce337a7ee4400d8, entries=150, sequenceid=366, filesize=11.9 K 2023-04-05 18:55:53,110 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/1bcc2d521fb048a5b1378074a08e200a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/1bcc2d521fb048a5b1378074a08e200a 2023-04-05 18:55:53,114 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/1bcc2d521fb048a5b1378074a08e200a, entries=150, sequenceid=366, filesize=11.9 K 2023-04-05 18:55:53,115 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/67760f112f0640978b07f36b5a9c0ab7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/67760f112f0640978b07f36b5a9c0ab7 2023-04-05 18:55:53,117 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/67760f112f0640978b07f36b5a9c0ab7, entries=150, sequenceid=366, filesize=11.9 K 2023-04-05 18:55:53,118 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 6e819dddd14a04b6eb4e71fe4e02a010 in 62ms, sequenceid=366, compaction requested=false 2023-04-05 18:55:53,118 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:53,118 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:53,119 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,119 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,120 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,120 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:53,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e1a3733[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:53,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:53,123 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,124 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:53,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:53,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:53,125 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,125 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:53,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:53,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,126 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:53,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:53,129 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:53,129 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:53,129 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:53,129 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:53,129 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:53,130 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,130 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,130 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,132 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,134 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,134 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,134 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,134 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:53,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:53,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,134 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,135 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:53,135 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:53,135 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,135 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,135 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,135 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,235 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:53,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:53,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:53,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:53,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:53,238 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:53,238 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,238 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:53,238 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:53,239 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,239 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,241 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,241 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,241 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,241 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:53,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:53,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:53,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:53,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:53,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:53,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:53,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:53,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:53,243 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,243 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:53,243 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:53,243 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:53,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:53,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:53,243 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:53,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,245 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,245 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,245 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:53,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1baed4bc[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:53,246 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:53,246 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,247 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:53,247 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:53,247 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:53,248 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,248 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:53,249 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:53,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:53,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:53,251 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:53,251 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:53,251 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:53,251 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:53,251 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:53,253 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,253 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,254 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,254 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,257 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,258 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:53,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:53,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:53,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,258 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:53,258 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:53,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,258 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:53,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:55:53,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:53,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:53,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:53,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:53,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:53,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:53,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/51da5d328f2c49779bb588ef932d0e70 2023-04-05 18:55:53,358 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:53,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:53,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:53,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:53,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:53,362 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:53,362 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,362 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:53,362 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:53,362 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,363 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,364 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:53,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:53,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:53,365 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,365 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:53,365 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:53,365 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:53,367 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7a75a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:53,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:53,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:53,371 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,371 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,372 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:53,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:53,375 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:53,375 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:53,375 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:53,375 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:53,375 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:53,376 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,376 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,376 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,378 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,380 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:53,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:53,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:53,380 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,381 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:53,381 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:53,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721013374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720953370, max timestamp=1680720953381Num uniques 859; 2023-04-05 18:55:53,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:53,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720953370, max timestamp=1680720953381Num uniques 859; 2023-04-05 18:55:53,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:53,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720953370, max timestamp=1680720953381Num uniques 859; 2023-04-05 18:55:53,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:53,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721013401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721013407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721013408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721013409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,481 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:53,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:53,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:53,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:53,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:53,484 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:53,484 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,484 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:53,484 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:53,485 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,485 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,487 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,487 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,487 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,487 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:53,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:53,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:53,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:53,488 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:53,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:53,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721013483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,490 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,490 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,490 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,490 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:53,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e8af57f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:53,491 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:53,491 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,492 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,493 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,493 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:53,493 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:53,493 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:53,493 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:53,494 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,494 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:53,494 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,494 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:53,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:53,496 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:53,496 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:53,496 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:53,496 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:53,496 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:53,499 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,499 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,499 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,499 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,500 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,502 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,502 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,502 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,502 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,502 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:53,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:53,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:53,502 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,502 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,502 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,502 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,502 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:53,503 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,503 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:53,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721013512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721013514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721013517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721013517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,603 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:53,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:53,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:53,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:53,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:53,606 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:53,606 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,606 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:53,606 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:53,606 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,606 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,607 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,607 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,608 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,608 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:53,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:53,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:53,609 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,609 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:53,609 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:53,609 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:53,611 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3166f632[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:53,613 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:53,613 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,615 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:53,615 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,615 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:53,616 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:53,616 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,616 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:53,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:53,619 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:53,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:53,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:53,619 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:53,619 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:53,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,622 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,623 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,624 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,624 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,624 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:53,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:53,624 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,625 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,625 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,625 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,625 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,625 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,625 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,625 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:53,625 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:53,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721013691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721013718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721013718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721013721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:53,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721013722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,725 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:53,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:53,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:53,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:53,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:53,728 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:53,728 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,728 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:53,728 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:53,729 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,729 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,731 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,731 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,731 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,731 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:53,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:53,733 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,733 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:53,733 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:53,733 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:53,734 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c80cf96[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:53,736 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:53,736 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,737 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:53,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,738 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:53,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:53,741 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:53,741 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:53,741 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:53,741 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:53,741 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:53,746 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,746 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,746 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,747 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,748 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,749 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,750 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,750 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,750 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,750 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:53,750 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,750 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:53,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:53,750 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,750 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,750 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,750 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:53,750 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:53,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/4f1b29773c1845fdb22db0d31a4f78f8 2023-04-05 18:55:53,850 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:53,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:53,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:53,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:53,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:53,853 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:53,853 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,853 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:53,853 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:53,854 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,854 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,855 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,855 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,855 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,855 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:53,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:53,857 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,857 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:53,857 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:53,857 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:53,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,858 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,858 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64657e2b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:53,860 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:53,860 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,862 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:53,862 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,862 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,864 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:53,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:53,866 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:53,866 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:53,866 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:53,866 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:53,866 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:53,868 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,868 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,868 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,868 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,871 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,871 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,871 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,871 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:53,871 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,871 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,872 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,872 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,872 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,872 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:53,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,872 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:53,872 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:53,972 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:53,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:53,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:53,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:53,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:53,975 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:53,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:53,975 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:53,976 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,976 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,978 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,978 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,978 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,978 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:53,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:53,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:53,979 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,980 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:53,980 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:53,980 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:53,981 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:53,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d6d4ed5[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:53,983 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:53,983 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,984 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,985 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:53,985 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,985 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:53,985 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:53,985 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:53,986 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:53,986 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,986 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,986 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:53,989 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:53,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:53,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:53,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:53,989 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:53,992 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,992 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,992 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:53,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:53,993 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,996 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:53,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,999 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:53,999 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:53,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:53,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:53,999 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:53,999 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:53,999 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:53,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:53,999 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:53,999 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:53,999 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,999 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:53,999 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:53,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721013997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721014024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721014025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721014025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721014027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,099 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:54,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:54,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:54,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:54,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:54,102 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:54,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:54,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:54,103 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,103 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,110 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:54,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:54,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:54,112 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,112 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:54,112 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:54,112 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:54,114 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,114 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:54,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bbce62c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:54,115 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:54,116 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,119 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:54,120 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,120 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:54,121 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:54,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:54,124 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:54,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:54,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:54,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:54,124 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:54,125 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,125 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,125 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,125 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,127 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,133 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,133 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,133 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,133 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,133 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,133 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:54,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:54,133 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,133 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,133 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:54,133 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,133 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,133 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:54,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/00181c70dcb4454084f9c4e9f2174702 2023-04-05 18:55:54,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/51da5d328f2c49779bb588ef932d0e70 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/51da5d328f2c49779bb588ef932d0e70 2023-04-05 18:55:54,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/51da5d328f2c49779bb588ef932d0e70, entries=150, sequenceid=380, filesize=11.9 K 2023-04-05 18:55:54,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/4f1b29773c1845fdb22db0d31a4f78f8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/4f1b29773c1845fdb22db0d31a4f78f8 2023-04-05 18:55:54,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/4f1b29773c1845fdb22db0d31a4f78f8, entries=150, sequenceid=380, filesize=11.9 K 2023-04-05 18:55:54,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/00181c70dcb4454084f9c4e9f2174702 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/00181c70dcb4454084f9c4e9f2174702 2023-04-05 18:55:54,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/00181c70dcb4454084f9c4e9f2174702, entries=150, sequenceid=380, filesize=11.9 K 2023-04-05 18:55:54,187 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6e819dddd14a04b6eb4e71fe4e02a010 in 842ms, sequenceid=380, compaction requested=true 2023-04-05 18:55:54,187 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:54,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:54,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:55:54,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-05 18:55:54,187 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:54,188 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:54,192 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:54,192 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:54,192 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:55:54,192 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:55:54,192 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,192 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,192 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/93b0df5adc734e82b84ad399c052b390, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/67760f112f0640978b07f36b5a9c0ab7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/00181c70dcb4454084f9c4e9f2174702] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=36.5 K 2023-04-05 18:55:54,192 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/07fb969d0072401997e9c6e976ba2d3b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/85a61d3baa174f8a9ce337a7ee4400d8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/51da5d328f2c49779bb588ef932d0e70] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=36.5 K 2023-04-05 18:55:54,193 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 93b0df5adc734e82b84ad399c052b390, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680720951867 2023-04-05 18:55:54,193 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 07fb969d0072401997e9c6e976ba2d3b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680720951867 2023-04-05 18:55:54,193 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 67760f112f0640978b07f36b5a9c0ab7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1680720952210 2023-04-05 18:55:54,193 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 85a61d3baa174f8a9ce337a7ee4400d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1680720952210 2023-04-05 18:55:54,194 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 51da5d328f2c49779bb588ef932d0e70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680720953339 2023-04-05 18:55:54,194 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 00181c70dcb4454084f9c4e9f2174702, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680720953339 2023-04-05 18:55:54,217 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:54,219 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:54,234 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:54,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:54,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:54,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:54,245 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:54,245 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,245 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:54,245 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:54,246 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,246 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,248 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,248 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:54,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:54,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:54,249 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,249 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:54,249 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-05 18:55:54,249 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:54,249 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:54,249 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:54,250 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:54,250 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:54,250 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:54,251 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/4830da92bc6b437d805dc1a08f78fe6f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4830da92bc6b437d805dc1a08f78fe6f 2023-04-05 18:55:54,256 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into 4830da92bc6b437d805dc1a08f78fe6f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:54,256 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:54,256 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=13, startTime=1680720954187; duration=0sec 2023-04-05 18:55:54,256 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:54,256 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:54,257 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:54,257 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:55:54,257 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,257 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/de5b57497fbd49ed9c781e775caa9165, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/1bcc2d521fb048a5b1378074a08e200a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/4f1b29773c1845fdb22db0d31a4f78f8] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=36.5 K 2023-04-05 18:55:54,257 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting de5b57497fbd49ed9c781e775caa9165, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680720951867 2023-04-05 18:55:54,258 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1bcc2d521fb048a5b1378074a08e200a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1680720952210 2023-04-05 18:55:54,258 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4f1b29773c1845fdb22db0d31a4f78f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680720953339 2023-04-05 18:55:54,259 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/82679c5d12574f1893657c9f3234902f 2023-04-05 18:55:54,269 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#476 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:54,276 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/af5788f5535549ce974a8319ffdb0432 2023-04-05 18:55:54,289 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/68944992e0b14fe6bd2871b839553035 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/68944992e0b14fe6bd2871b839553035 2023-04-05 18:55:54,294 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into 68944992e0b14fe6bd2871b839553035(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:54,294 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:54,294 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=13, startTime=1680720954187; duration=0sec 2023-04-05 18:55:54,294 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:54,330 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/b65fbec8c20647199804fca4280f4a49 2023-04-05 18:55:54,334 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/82679c5d12574f1893657c9f3234902f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/82679c5d12574f1893657c9f3234902f 2023-04-05 18:55:54,337 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/82679c5d12574f1893657c9f3234902f, entries=150, sequenceid=416, filesize=11.9 K 2023-04-05 18:55:54,337 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/af5788f5535549ce974a8319ffdb0432 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/af5788f5535549ce974a8319ffdb0432 2023-04-05 18:55:54,341 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/af5788f5535549ce974a8319ffdb0432, entries=150, sequenceid=416, filesize=11.9 K 2023-04-05 18:55:54,342 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/b65fbec8c20647199804fca4280f4a49 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b65fbec8c20647199804fca4280f4a49 2023-04-05 18:55:54,356 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b65fbec8c20647199804fca4280f4a49, entries=150, sequenceid=416, filesize=11.9 K 2023-04-05 18:55:54,357 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 6e819dddd14a04b6eb4e71fe4e02a010 in 108ms, sequenceid=416, compaction requested=false 2023-04-05 18:55:54,357 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:54,357 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,357 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:54,357 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:54,357 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,357 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:54,357 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:54,359 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,359 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,359 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,359 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:54,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119a7cb2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:54,361 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:54,361 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,362 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:54,362 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,362 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:54,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,365 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,365 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:54,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:54,368 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:54,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:54,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:54,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:54,369 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:54,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,371 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,372 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:54,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:54,375 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,374 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,375 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,375 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,375 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,375 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:54,375 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:54,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,375 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:54,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,375 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,475 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:54,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:54,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:54,479 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:54,479 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,479 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:54,479 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:54,479 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,479 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,480 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,480 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:54,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:54,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:54,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:54,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:54,486 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,486 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:54,486 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:54,486 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:54,488 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,489 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ec1cc8c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:54,505 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:54,505 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,507 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,507 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:54,507 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,507 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:54,507 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:54,507 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:54,510 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:54,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:54,514 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:54,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:54,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:54,514 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:54,514 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:54,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,516 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,516 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,517 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,520 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,520 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,521 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,521 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,521 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,521 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:54,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:54,521 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,521 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,521 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,521 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:54,521 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:54,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:54,521 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:54,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:55:54,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:54,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:54,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:54,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:54,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:54,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:54,535 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/aa5fe798fa524b1585be55f87cea5fc1 2023-04-05 18:55:54,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/c19a90c94be047aa84a1fa3e2fdb125b 2023-04-05 18:55:54,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720954567, max timestamp=1680720954580Num uniques 859; 2023-04-05 18:55:54,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:54,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720954567, max timestamp=1680720954580Num uniques 859; 2023-04-05 18:55:54,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:54,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720954567, max timestamp=1680720954580Num uniques 859; 2023-04-05 18:55:54,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:55:54,621 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:54,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:54,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:54,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:54,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:54,624 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:54,624 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,624 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:54,624 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:54,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,626 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,626 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,626 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,627 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:54,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:54,628 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,628 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:54,628 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:54,628 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:54,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,630 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:54,630 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:54,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@797d2a67[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:54,633 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/e6ca6fca4dcc4f728d8805ddb6381d3e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e6ca6fca4dcc4f728d8805ddb6381d3e 2023-04-05 18:55:54,632 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:54,633 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,634 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:54,634 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:54,634 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:54,634 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,634 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:54,636 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:54,636 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,636 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,636 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,638 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into e6ca6fca4dcc4f728d8805ddb6381d3e(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:54,638 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:54,638 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=13, startTime=1680720954187; duration=0sec 2023-04-05 18:55:54,638 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:54,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:54,641 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:54,641 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:54,641 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:54,641 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:54,641 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:54,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721014632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721014633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721014635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,643 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,648 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,644 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721014636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721014644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,656 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:54,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,656 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:54,657 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:54,656 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,657 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721014745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721014745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721014745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,757 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:54,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:54,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:54,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721014750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:54,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:54,760 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:54,760 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,760 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:54,760 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:54,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,761 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721014754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,764 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,764 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,764 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,764 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:54,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:54,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:54,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:54,769 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:54,770 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,770 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:54,770 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:54,770 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:54,772 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:54,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55dc2ee2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:54,774 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:54,774 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,775 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:54,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,777 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:54,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:54,779 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:54,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:54,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:54,779 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:54,779 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:54,781 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,781 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,782 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,782 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,783 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,785 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:54,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,786 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:54,786 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:54,786 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,886 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:54,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:54,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:54,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:54,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:54,889 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:54,889 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,889 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:54,889 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:54,890 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,890 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,891 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,891 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:54,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:54,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:54,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:54,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:54,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:54,893 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,893 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:54,893 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:54,893 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:54,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:54,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:54,893 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:54,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:54,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1761b707[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:54,896 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:54,896 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,898 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:54,898 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,898 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:54,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:54,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,899 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,899 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:54,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:54,901 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:54,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:54,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:54,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:54,901 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:54,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:54,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,903 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:54,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,904 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:54,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:54,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:54,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:54,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:54,907 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:54,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:54,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,907 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:54,907 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:54,908 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:54,907 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:54,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:54,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721014953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721014953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721014954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/e6220b7500084c02a57df45d0ed5ab5b 2023-04-05 18:55:54,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721014961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/aa5fe798fa524b1585be55f87cea5fc1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/aa5fe798fa524b1585be55f87cea5fc1 2023-04-05 18:55:54,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/aa5fe798fa524b1585be55f87cea5fc1, entries=150, sequenceid=429, filesize=11.9 K 2023-04-05 18:55:54,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:54,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/c19a90c94be047aa84a1fa3e2fdb125b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/c19a90c94be047aa84a1fa3e2fdb125b 2023-04-05 18:55:54,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721014965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:54,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/c19a90c94be047aa84a1fa3e2fdb125b, entries=150, sequenceid=429, filesize=11.9 K 2023-04-05 18:55:54,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/e6220b7500084c02a57df45d0ed5ab5b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e6220b7500084c02a57df45d0ed5ab5b 2023-04-05 18:55:54,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e6220b7500084c02a57df45d0ed5ab5b, entries=150, sequenceid=429, filesize=11.9 K 2023-04-05 18:55:54,979 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6e819dddd14a04b6eb4e71fe4e02a010 in 450ms, sequenceid=429, compaction requested=true 2023-04-05 18:55:54,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:54,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:54,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:54,979 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:54,979 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:54,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:54,980 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:54,980 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:54,980 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:55:54,980 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:55:54,980 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,980 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:54,980 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e6ca6fca4dcc4f728d8805ddb6381d3e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/82679c5d12574f1893657c9f3234902f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/aa5fe798fa524b1585be55f87cea5fc1] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=36.6 K 2023-04-05 18:55:54,980 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/68944992e0b14fe6bd2871b839553035, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/af5788f5535549ce974a8319ffdb0432, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/c19a90c94be047aa84a1fa3e2fdb125b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=36.6 K 2023-04-05 18:55:54,980 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting e6ca6fca4dcc4f728d8805ddb6381d3e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680720953339 2023-04-05 18:55:54,981 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 68944992e0b14fe6bd2871b839553035, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680720953339 2023-04-05 18:55:54,981 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 82679c5d12574f1893657c9f3234902f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680720953399 2023-04-05 18:55:54,981 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting af5788f5535549ce974a8319ffdb0432, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680720953399 2023-04-05 18:55:54,981 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting aa5fe798fa524b1585be55f87cea5fc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680720954512 2023-04-05 18:55:54,981 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting c19a90c94be047aa84a1fa3e2fdb125b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680720954512 2023-04-05 18:55:54,986 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#482 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:54,986 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:55,008 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:55,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:55,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:55,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:55,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:55,010 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:55,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:55,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:55,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:55,011 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:55,011 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:55,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:55,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:55,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:55,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:55,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:55,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:55,015 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:55,015 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:55,015 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-05 18:55:55,015 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:55,015 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:55,015 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:55,015 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:55,015 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:55,015 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:55,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:55,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:55,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721015271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721015273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721015274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721015276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721015277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721015382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721015382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721015386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721015390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,400 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/90e0413eff404d3a8199cd8255da62a6 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/90e0413eff404d3a8199cd8255da62a6 2023-04-05 18:55:55,403 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/0e897b574f994af6bbd31a0749cacfa8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/0e897b574f994af6bbd31a0749cacfa8 2023-04-05 18:55:55,405 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into 90e0413eff404d3a8199cd8255da62a6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:55,405 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:55,405 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=13, startTime=1680720954979; duration=0sec 2023-04-05 18:55:55,405 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:55,405 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:55,406 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:55,406 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:55:55,406 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:55,406 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4830da92bc6b437d805dc1a08f78fe6f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b65fbec8c20647199804fca4280f4a49, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e6220b7500084c02a57df45d0ed5ab5b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=36.6 K 2023-04-05 18:55:55,407 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4830da92bc6b437d805dc1a08f78fe6f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680720953339 2023-04-05 18:55:55,407 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting b65fbec8c20647199804fca4280f4a49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680720953399 2023-04-05 18:55:55,407 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into 0e897b574f994af6bbd31a0749cacfa8(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:55,407 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:55,407 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=13, startTime=1680720954979; duration=0sec 2023-04-05 18:55:55,407 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e6220b7500084c02a57df45d0ed5ab5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680720954512 2023-04-05 18:55:55,407 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:55,412 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#485 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:55,421 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/a80754f71c654b8bbc53ecabe36f0867 2023-04-05 18:55:55,426 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/d737f9d471904e748411a9f341e0f81f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d737f9d471904e748411a9f341e0f81f 2023-04-05 18:55:55,430 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into d737f9d471904e748411a9f341e0f81f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:55,430 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:55,430 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=13, startTime=1680720954979; duration=0sec 2023-04-05 18:55:55,430 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:55,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721015590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721015591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721015598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721015598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721015786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,834 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/222f8dda7157443e852584a4dcbfa7c9 2023-04-05 18:55:55,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721015895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721015897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721015902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:55,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:55,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721015902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,259 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/e7e8306a61264d1f8ce00627b62eb572 2023-04-05 18:55:56,263 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/a80754f71c654b8bbc53ecabe36f0867 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80754f71c654b8bbc53ecabe36f0867 2023-04-05 18:55:56,265 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80754f71c654b8bbc53ecabe36f0867, entries=150, sequenceid=467, filesize=11.9 K 2023-04-05 18:55:56,266 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/222f8dda7157443e852584a4dcbfa7c9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/222f8dda7157443e852584a4dcbfa7c9 2023-04-05 18:55:56,269 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/222f8dda7157443e852584a4dcbfa7c9, entries=150, sequenceid=467, filesize=11.9 K 2023-04-05 18:55:56,270 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/e7e8306a61264d1f8ce00627b62eb572 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e7e8306a61264d1f8ce00627b62eb572 2023-04-05 18:55:56,273 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e7e8306a61264d1f8ce00627b62eb572, entries=150, sequenceid=467, filesize=11.9 K 2023-04-05 18:55:56,273 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 6e819dddd14a04b6eb4e71fe4e02a010 in 1258ms, sequenceid=467, compaction requested=false 2023-04-05 18:55:56,274 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:56,274 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:56,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:56,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:56,274 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:56,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,275 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,276 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,276 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:56,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4714accb[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:56,277 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:56,277 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,278 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,279 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:56,279 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:56,279 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:56,279 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,279 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:56,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:56,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:56,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:56,282 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:56,282 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:56,282 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:56,282 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:56,282 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:56,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,285 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:56,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,287 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,289 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,289 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,289 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,289 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:56,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:56,289 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:56,289 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:56,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:56,290 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:56,290 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:56,290 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:56,289 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:56,290 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,290 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:56,390 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:56,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:56,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:56,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:56,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:56,393 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:56,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:56,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:56,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:56,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,395 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:56,395 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:56,395 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,395 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:56,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:56,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:56,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:56,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:56,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:56,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:56,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:56,397 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,397 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:56,397 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:55:56,397 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:56,397 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:56,397 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:56,397 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:56,397 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:56,398 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:56,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:56,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:56,407 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/d72f9ab20e1e4cddb765f6e9bf876a77 2023-04-05 18:55:56,419 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/f4ffd644f57446d29dcf48814a0c2515 2023-04-05 18:55:56,431 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/1cabe40a0e2a4959869a030ad9bad33f 2023-04-05 18:55:56,435 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/d72f9ab20e1e4cddb765f6e9bf876a77 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d72f9ab20e1e4cddb765f6e9bf876a77 2023-04-05 18:55:56,437 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720956424, max timestamp=1680720956436Num uniques 859; 2023-04-05 18:55:56,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:56,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720956424, max timestamp=1680720956436Num uniques 859; 2023-04-05 18:55:56,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:56,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720956424, max timestamp=1680720956436Num uniques 859; 2023-04-05 18:55:56,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:55:56,439 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d72f9ab20e1e4cddb765f6e9bf876a77, entries=150, sequenceid=480, filesize=11.9 K 2023-04-05 18:55:56,440 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/f4ffd644f57446d29dcf48814a0c2515 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f4ffd644f57446d29dcf48814a0c2515 2023-04-05 18:55:56,449 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f4ffd644f57446d29dcf48814a0c2515, entries=150, sequenceid=480, filesize=11.9 K 2023-04-05 18:55:56,449 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/1cabe40a0e2a4959869a030ad9bad33f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1cabe40a0e2a4959869a030ad9bad33f 2023-04-05 18:55:56,452 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1cabe40a0e2a4959869a030ad9bad33f, entries=150, sequenceid=480, filesize=11.9 K 2023-04-05 18:55:56,453 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=67.09 KB/68700 for 6e819dddd14a04b6eb4e71fe4e02a010 in 56ms, sequenceid=480, compaction requested=true 2023-04-05 18:55:56,453 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:56,453 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:56,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:56,457 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=73.80 KB heapSize=201 KB 2023-04-05 18:55:56,458 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,458 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,458 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,458 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:56,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:56,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:56,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:56,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:56,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:56,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57aa985c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:56,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:56,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:56,462 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,462 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:56,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:56,463 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:56,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/b86ade27a1704a929e4c21ba28092f0b 2023-04-05 18:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:56,467 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:56,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:56,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:56,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:56,467 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:56,469 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,469 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,469 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,470 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,472 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,473 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,474 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,474 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,474 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,474 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:56,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:56,474 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:56,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:56,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:56,474 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:56,474 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:56,474 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:56,474 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,474 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:56,474 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,475 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/acfeea9deb3e45659d27c95a290b31e3 2023-04-05 18:55:56,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:56,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721016489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721016491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721016492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721016499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,574 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:56,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:56,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:56,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:56,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:56,577 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:56,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:56,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:56,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:56,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,580 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:56,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:56,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:56,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:56,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:56,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:56,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:56,582 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,582 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:56,582 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:56,582 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:56,583 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,583 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:56,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b51775[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:56,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:56,585 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,586 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:56,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:56,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:56,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:56,588 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,588 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:56,588 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:56,588 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:56,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:56,590 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:56,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:56,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:56,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:56,590 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:56,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:56,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:56,593 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,594 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:56,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:56,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:56,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:56,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:56,596 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:56,596 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,597 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:56,597 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:56,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:56,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:56,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721016599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721016602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721016607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721016608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,697 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:56,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:56,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:56,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:56,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:56,700 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:56,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:56,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:56,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:56,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:56,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:56,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:56,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:56,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:56,703 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,703 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:56,703 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:56,703 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:56,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:56,705 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,705 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:56,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79f64d8b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:56,706 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:56,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:56,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:56,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:56,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:56,712 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:56,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:56,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:56,713 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:56,713 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:56,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:56,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,716 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,718 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,718 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,718 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,718 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:56,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:56,718 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:56,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:56,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:56,718 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,719 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,719 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:56,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:56,719 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,719 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:56,719 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:56,719 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:56,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721016800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721016804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721016808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721016812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,819 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:56,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:56,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:56,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721016813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:56,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:56,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:56,822 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:56,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:56,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:56,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:56,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:56,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:56,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:56,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:56,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:56,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:56,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:56,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:56,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:56,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:56,826 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,826 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:56,826 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:56,826 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,827 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:56,827 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:56,827 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,827 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:56,827 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:56,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,829 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,829 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,829 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b66d48a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:56,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:56,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:56,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:56,835 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:56,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:56,837 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:56,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:56,839 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:56,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:56,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:56,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:56,839 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:56,841 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,841 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,841 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:56,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:56,842 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,843 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:56,846 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:56,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:56,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:56,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:56,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:56,846 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,847 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,847 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,847 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:56,847 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:56,847 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,847 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:56,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:56,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/648245ea3bb946a18d4a20e133572080 2023-04-05 18:55:56,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/b86ade27a1704a929e4c21ba28092f0b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b86ade27a1704a929e4c21ba28092f0b 2023-04-05 18:55:56,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b86ade27a1704a929e4c21ba28092f0b, entries=200, sequenceid=508, filesize=14.3 K 2023-04-05 18:55:56,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/acfeea9deb3e45659d27c95a290b31e3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/acfeea9deb3e45659d27c95a290b31e3 2023-04-05 18:55:56,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/acfeea9deb3e45659d27c95a290b31e3, entries=150, sequenceid=508, filesize=11.9 K 2023-04-05 18:55:56,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/648245ea3bb946a18d4a20e133572080 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/648245ea3bb946a18d4a20e133572080 2023-04-05 18:55:56,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/648245ea3bb946a18d4a20e133572080, entries=150, sequenceid=508, filesize=11.9 K 2023-04-05 18:55:56,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~220.88 KB/226176, currentSize=100.63 KB/103050 for 6e819dddd14a04b6eb4e71fe4e02a010 in 445ms, sequenceid=508, compaction requested=true 2023-04-05 18:55:56,902 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:56,902 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:56,902 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:56,902 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:56,902 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:56,902 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:56,904 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:56,904 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:55:56,904 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,904 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/90e0413eff404d3a8199cd8255da62a6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/222f8dda7157443e852584a4dcbfa7c9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f4ffd644f57446d29dcf48814a0c2515, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/acfeea9deb3e45659d27c95a290b31e3] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=48.6 K 2023-04-05 18:55:56,904 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52240 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:56,905 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 90e0413eff404d3a8199cd8255da62a6, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680720954512 2023-04-05 18:55:56,905 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:55:56,905 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,905 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/0e897b574f994af6bbd31a0749cacfa8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80754f71c654b8bbc53ecabe36f0867, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d72f9ab20e1e4cddb765f6e9bf876a77, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b86ade27a1704a929e4c21ba28092f0b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=51.0 K 2023-04-05 18:55:56,906 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 222f8dda7157443e852584a4dcbfa7c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720954632 2023-04-05 18:55:56,906 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 0e897b574f994af6bbd31a0749cacfa8, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680720954512 2023-04-05 18:55:56,906 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting f4ffd644f57446d29dcf48814a0c2515, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680720955270 2023-04-05 18:55:56,907 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting a80754f71c654b8bbc53ecabe36f0867, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720954632 2023-04-05 18:55:56,907 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting acfeea9deb3e45659d27c95a290b31e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680720956449 2023-04-05 18:55:56,907 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting d72f9ab20e1e4cddb765f6e9bf876a77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680720955270 2023-04-05 18:55:56,908 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting b86ade27a1704a929e4c21ba28092f0b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680720956447 2023-04-05 18:55:56,913 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:56,918 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#495 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:56,923 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/154252475c874d52b756d0488414379d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/154252475c874d52b756d0488414379d 2023-04-05 18:55:56,927 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into 154252475c874d52b756d0488414379d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:56,927 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:56,927 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=12, startTime=1680720956902; duration=0sec 2023-04-05 18:55:56,927 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:56,927 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:56,928 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:56,928 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:55:56,928 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,928 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d737f9d471904e748411a9f341e0f81f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e7e8306a61264d1f8ce00627b62eb572, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1cabe40a0e2a4959869a030ad9bad33f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/648245ea3bb946a18d4a20e133572080] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=48.6 K 2023-04-05 18:55:56,929 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting d737f9d471904e748411a9f341e0f81f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680720954512 2023-04-05 18:55:56,930 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e7e8306a61264d1f8ce00627b62eb572, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720954632 2023-04-05 18:55:56,931 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1cabe40a0e2a4959869a030ad9bad33f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680720955270 2023-04-05 18:55:56,931 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 648245ea3bb946a18d4a20e133572080, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680720956449 2023-04-05 18:55:56,941 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#496 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:56,947 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:56,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:56,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:56,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:56,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:56,950 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:56,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:56,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:56,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:56,951 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,951 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,952 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:56,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,953 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:56,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:56,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:56,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:56,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:56,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:56,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:56,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:56,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:56,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:56,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:56,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:56,955 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 18:55:56,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:56,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:56,956 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:56,956 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:56,956 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:56,956 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:56,972 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/339fb210ea00482fbc973aeb04840024 2023-04-05 18:55:57,008 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/13c77581ee6448f493927af48623163a 2023-04-05 18:55:57,022 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/85f110b32e0143aa8ee37441bcc36a40 2023-04-05 18:55:57,026 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/339fb210ea00482fbc973aeb04840024 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/339fb210ea00482fbc973aeb04840024 2023-04-05 18:55:57,031 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/339fb210ea00482fbc973aeb04840024, entries=150, sequenceid=527, filesize=11.9 K 2023-04-05 18:55:57,032 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/13c77581ee6448f493927af48623163a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/13c77581ee6448f493927af48623163a 2023-04-05 18:55:57,036 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/13c77581ee6448f493927af48623163a, entries=150, sequenceid=527, filesize=11.9 K 2023-04-05 18:55:57,037 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/85f110b32e0143aa8ee37441bcc36a40 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/85f110b32e0143aa8ee37441bcc36a40 2023-04-05 18:55:57,055 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/85f110b32e0143aa8ee37441bcc36a40, entries=150, sequenceid=527, filesize=11.9 K 2023-04-05 18:55:57,056 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 6e819dddd14a04b6eb4e71fe4e02a010 in 101ms, sequenceid=527, compaction requested=false 2023-04-05 18:55:57,056 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:57,056 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:57,066 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,066 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,067 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,067 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188dc7c2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:57,071 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:57,071 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,072 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:57,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:57,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:57,073 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,073 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:57,074 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,075 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:57,075 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,075 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:57,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:57,080 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:57,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:57,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:57,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:57,080 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:57,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,088 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,088 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,089 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,089 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,089 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,089 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,089 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:57,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:57,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,089 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:57,089 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:57,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:57,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:55:57,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:57,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:57,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:57,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:57,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:57,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:57,136 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/67c0bc99cc8545758ca27d81202ebc0e 2023-04-05 18:55:57,145 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/77277dd98c834d15ba897fdd3d1c691b 2023-04-05 18:55:57,157 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/6e5b90ff8d9344faa8ab30b0b66f5144 2023-04-05 18:55:57,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/67c0bc99cc8545758ca27d81202ebc0e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/67c0bc99cc8545758ca27d81202ebc0e 2023-04-05 18:55:57,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/67c0bc99cc8545758ca27d81202ebc0e, entries=250, sequenceid=537, filesize=16.7 K 2023-04-05 18:55:57,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/77277dd98c834d15ba897fdd3d1c691b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/77277dd98c834d15ba897fdd3d1c691b 2023-04-05 18:55:57,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/77277dd98c834d15ba897fdd3d1c691b, entries=100, sequenceid=537, filesize=9.5 K 2023-04-05 18:55:57,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/6e5b90ff8d9344faa8ab30b0b66f5144 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/6e5b90ff8d9344faa8ab30b0b66f5144 2023-04-05 18:55:57,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/6e5b90ff8d9344faa8ab30b0b66f5144, entries=100, sequenceid=537, filesize=9.5 K 2023-04-05 18:55:57,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=73.80 KB/75570 for 6e819dddd14a04b6eb4e71fe4e02a010 in 50ms, sequenceid=537, compaction requested=true 2023-04-05 18:55:57,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:57,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:57,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:55:57,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-05 18:55:57,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:57,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 18:55:57,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:57,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:57,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:57,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:57,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:57,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:57,189 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:57,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:57,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:57,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:57,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:57,192 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:57,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:57,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:57,193 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,193 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,194 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,194 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,194 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:57,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:57,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:57,196 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,196 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:57,196 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:57,196 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:57,199 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:57,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53486519[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:57,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:57,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,201 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:57,203 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:57,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:57,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:57,205 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:57,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:57,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:57,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:57,205 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:57,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,208 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721017198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,211 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,211 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,211 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:57,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:57,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:57,212 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:57,212 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:57,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721017207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721017208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721017209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,312 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:57,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:57,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:57,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:57,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:57,315 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:57,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:57,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:57,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,317 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,317 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:57,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721017312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:57,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:57,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:57,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:57,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:57,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:57,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:57,319 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:57,320 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,321 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:57,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721017313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1899e0ee[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:57,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:57,324 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,325 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:57,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:57,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:57,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:57,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721017319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721017320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,329 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/05779f7bf7b94a6ebae226b060ac5ca0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/05779f7bf7b94a6ebae226b060ac5ca0 2023-04-05 18:55:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:57,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,334 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into 05779f7bf7b94a6ebae226b060ac5ca0(size=13.1 K), total size for store is 41.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:57,335 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:57,335 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=12, startTime=1680720956902; duration=0sec 2023-04-05 18:55:57,335 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-05 18:55:57,335 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-05 18:55:57,335 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-05 18:55:57,335 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-05 18:55:57,335 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. because compaction request was cancelled 2023-04-05 18:55:57,336 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:57,336 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:57,336 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:57,336 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:57,336 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:57,336 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:57,337 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42626 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:57,337 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:55:57,337 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,337 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/05779f7bf7b94a6ebae226b060ac5ca0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/339fb210ea00482fbc973aeb04840024, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/67c0bc99cc8545758ca27d81202ebc0e] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=41.6 K 2023-04-05 18:55:57,338 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 05779f7bf7b94a6ebae226b060ac5ca0, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680720956449 2023-04-05 18:55:57,338 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 339fb210ea00482fbc973aeb04840024, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1680720956490 2023-04-05 18:55:57,338 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,338 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,338 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,339 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 67c0bc99cc8545758ca27d81202ebc0e, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680720957116 2023-04-05 18:55:57,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,341 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,342 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:57,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:57,344 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,344 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:57,345 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:57,344 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,345 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,345 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#504 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:57,358 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/c82b697c0bcf4aa4b172297cb863c6bb as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/c82b697c0bcf4aa4b172297cb863c6bb 2023-04-05 18:55:57,363 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into c82b697c0bcf4aa4b172297cb863c6bb(size=13.1 K), total size for store is 34.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:57,363 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:57,363 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=12, startTime=1680720956902; duration=0sec 2023-04-05 18:55:57,363 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:57,363 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:55:57,364 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35306 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:55:57,364 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:55:57,364 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,364 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/154252475c874d52b756d0488414379d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/13c77581ee6448f493927af48623163a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/77277dd98c834d15ba897fdd3d1c691b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=34.5 K 2023-04-05 18:55:57,365 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 154252475c874d52b756d0488414379d, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680720956449 2023-04-05 18:55:57,365 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 13c77581ee6448f493927af48623163a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1680720956490 2023-04-05 18:55:57,365 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 77277dd98c834d15ba897fdd3d1c691b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680720957122 2023-04-05 18:55:57,371 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#505 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:57,393 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/b17c79badb9c468c829b40b8bc60a14f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b17c79badb9c468c829b40b8bc60a14f 2023-04-05 18:55:57,399 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into b17c79badb9c468c829b40b8bc60a14f(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:57,399 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:57,399 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=13, startTime=1680720957174; duration=0sec 2023-04-05 18:55:57,400 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:57,445 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:57,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:57,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:57,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:57,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:57,448 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:57,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:57,448 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:57,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,451 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:57,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:57,452 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,452 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:57,453 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:57,453 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:57,453 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:57,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3576a5f3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:57,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:57,456 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,459 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,459 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,459 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:57,460 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:57,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:57,464 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:57,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:57,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:57,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:57,464 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:57,466 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,468 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,470 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:57,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:57,471 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:57,471 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:57,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,472 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:57,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721017521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721017524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721017527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721017529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,571 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:57,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:57,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:57,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:57,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:57,574 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:57,574 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,574 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:57,574 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:57,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,576 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,576 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,576 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,576 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:57,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:57,578 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,578 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:57,578 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:57,578 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:57,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:57,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:57,578 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:57,579 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,579 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4333c748[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:57,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:57,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,583 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:57,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:57,585 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:57,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/7b401e7b19cf43d9b989c939cf76526c 2023-04-05 18:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:57,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:57,590 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:57,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:57,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:57,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:57,590 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:57,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,594 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,596 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,598 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:57,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:57,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,599 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,599 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:57,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:57,599 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:57,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,699 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:57,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:57,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:57,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:57,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:57,702 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:57,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:57,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:57,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,705 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,705 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:57,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:57,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:57,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:57,706 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,706 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:57,706 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:57,706 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5343129a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:57,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:57,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:57,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:57,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:57,712 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:57,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:57,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:57,712 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:57,712 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:57,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,714 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,717 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,721 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,721 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,721 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:57,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:57,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,721 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:57,722 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:57,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:57,759 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/ee6c46cd511549a1b119a6829fc9326d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/ee6c46cd511549a1b119a6829fc9326d 2023-04-05 18:55:57,762 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into ee6c46cd511549a1b119a6829fc9326d(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:57,762 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:57,763 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=13, startTime=1680720957174; duration=0sec 2023-04-05 18:55:57,763 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:57,822 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:57,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:57,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:57,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:57,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:57,825 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:57,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:57,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:57,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,826 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:57,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:57,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:57,828 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,828 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:57,828 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:57,828 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:57,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,830 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:57,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a068c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:57,832 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:57,832 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721017831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721017831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721017832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,834 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:57,834 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:57,834 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,834 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:57,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721017832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,834 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:57,834 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:57,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:57,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:57,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:57,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:57,838 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:57,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:57,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:57,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:57,839 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:57,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,841 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,842 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:57,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:57,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:55:57,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,844 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,844 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:57,844 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:57,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,844 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,944 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:57,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:57,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:57,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:57,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:57,947 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:57,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:57,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:57,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:57,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:57,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:57,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:57,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:57,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:57,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:57,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:57,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:57,952 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,952 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:57,952 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:57,952 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:57,952 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:57,952 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:57,952 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,952 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:57,952 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:57,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,953 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:57,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d362c3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:57,955 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:57,955 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:57,956 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,956 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:57,957 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:57,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:57,959 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:57,959 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:57,959 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:57,959 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:57,959 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:57,961 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,962 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,962 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:57,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,962 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,963 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:57,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:57,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:57,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:57,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:57,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:57,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:57,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:57,966 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:57,966 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:57,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:57,966 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:57,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:57,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/bb04e3bc0fa54e34865a86a914f33f92 2023-04-05 18:55:58,066 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:58,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:58,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:58,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:58,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:58,069 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:58,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:58,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:58,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:58,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,070 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,071 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:58,071 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:58,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:58,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:58,073 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:58,073 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:58,074 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:58,074 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:58,075 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:58,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5987fc96[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:58,079 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:58,079 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,081 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,081 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:58,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:58,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,083 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:58,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:58,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:58,086 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:58,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:58,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:58,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:58,086 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:58,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:58,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,088 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:58,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:58,089 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:58,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:58,095 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:58,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:58,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:58,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:58,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,095 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:58,095 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:58,095 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:58,195 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:58,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:58,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:58,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:58,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:58,198 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:58,198 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:58,198 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:58,198 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:58,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,200 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:58,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:58,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:58,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:58,201 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:58,201 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:58,202 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:58,202 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:58,203 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@129bf2fe[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:58,204 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:58,204 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,206 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:58,206 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:58,206 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:58,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:58,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,207 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:58,208 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:58,208 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:58,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:58,210 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:58,210 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:58,210 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:58,210 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:58,210 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:58,211 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,211 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:58,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,211 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:58,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:58,212 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,213 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,214 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,214 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,215 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,215 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:58,215 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:58,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:58,215 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:58,215 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:58,215 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:58,215 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:58,215 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:58,215 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,215 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:58,315 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:58,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:58,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:58,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:58,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:58,318 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:58,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:58,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:58,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:58,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,319 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,321 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:58,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:58,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:58,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:58,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:58,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:58,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:58,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:58,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:58,322 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:58,323 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:58,323 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:58,323 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:58,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:58,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:58,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:58,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:58,324 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:58,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:58,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46f1a847[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:58,326 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:58,326 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,327 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:58,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:58,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:58,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:58,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:58,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:58,330 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:58,330 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:58,330 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:58,330 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:58,330 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:58,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:58,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:58,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:58,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:58,333 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,334 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,336 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,336 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,336 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:58,336 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:58,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:58,337 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:58,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:58,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:58,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:58,337 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:58,337 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:58,337 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:58,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721018335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721018338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721018338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721018338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,406 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/0fa768bb1ea94edb8033eb9ca29ba17a 2023-04-05 18:55:58,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/7b401e7b19cf43d9b989c939cf76526c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/7b401e7b19cf43d9b989c939cf76526c 2023-04-05 18:55:58,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/7b401e7b19cf43d9b989c939cf76526c, entries=250, sequenceid=552, filesize=16.7 K 2023-04-05 18:55:58,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/bb04e3bc0fa54e34865a86a914f33f92 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bb04e3bc0fa54e34865a86a914f33f92 2023-04-05 18:55:58,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bb04e3bc0fa54e34865a86a914f33f92, entries=150, sequenceid=552, filesize=11.9 K 2023-04-05 18:55:58,417 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/0fa768bb1ea94edb8033eb9ca29ba17a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0fa768bb1ea94edb8033eb9ca29ba17a 2023-04-05 18:55:58,420 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0fa768bb1ea94edb8033eb9ca29ba17a, entries=150, sequenceid=552, filesize=11.9 K 2023-04-05 18:55:58,420 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 6e819dddd14a04b6eb4e71fe4e02a010 in 1243ms, sequenceid=552, compaction requested=true 2023-04-05 18:55:58,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:58,421 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:58,421 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-05 18:55:58,421 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:55:58,421 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:55:58,421 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:55:58,421 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-05 18:55:58,421 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-05 18:55:58,421 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. because compaction request was cancelled 2023-04-05 18:55:58,422 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-05 18:55:58,422 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-05 18:55:58,422 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-05 18:55:58,422 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47492 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:55:58,422 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. because compaction request was cancelled 2023-04-05 18:55:58,422 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:55:58,422 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:58,423 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/c82b697c0bcf4aa4b172297cb863c6bb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/85f110b32e0143aa8ee37441bcc36a40, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/6e5b90ff8d9344faa8ab30b0b66f5144, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0fa768bb1ea94edb8033eb9ca29ba17a] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=46.4 K 2023-04-05 18:55:58,423 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c82b697c0bcf4aa4b172297cb863c6bb, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680720956449 2023-04-05 18:55:58,423 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 85f110b32e0143aa8ee37441bcc36a40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1680720956490 2023-04-05 18:55:58,423 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6e5b90ff8d9344faa8ab30b0b66f5144, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680720957122 2023-04-05 18:55:58,424 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 0fa768bb1ea94edb8033eb9ca29ba17a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680720957168 2023-04-05 18:55:58,430 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#508 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:55:58,437 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/567aab9c3e594ed6b8c1a3120ad5e79d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/567aab9c3e594ed6b8c1a3120ad5e79d 2023-04-05 18:55:58,437 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:58,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:55:58,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:55:58,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:55:58,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:55:58,441 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:55:58,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:58,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:55:58,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:55:58,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,442 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into 567aab9c3e594ed6b8c1a3120ad5e79d(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:55:58,442 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:58,442 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=12, startTime=1680720958421; duration=0sec 2023-04-05 18:55:58,442 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:55:58,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:58,442 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:58,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:58,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:55:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:58,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:55:58,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:55:58,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:55:58,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:55:58,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:55:58,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:55:58,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:55:58,444 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:55:58,444 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:58,444 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. started... 2023-04-05 18:55:58,445 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-05 18:55:58,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:55:58,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:58,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:55:58,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:58,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:55:58,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:55:58,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:55:58,823 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. as already flushing 2023-04-05 18:55:58,854 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/bf85bceed9cf408b81864445cdcf24b2 2023-04-05 18:55:58,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:58,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721018893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:59,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721018999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:59,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721019204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,265 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/566d367afc27451e9453c077cbbe0c87 2023-04-05 18:55:59,277 DEBUG [Thread-3047] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x720d29f5 to 127.0.0.1:49755 2023-04-05 18:55:59,277 DEBUG [Thread-3047] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:59,279 DEBUG [Thread-3043] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65816b6e to 127.0.0.1:49755 2023-04-05 18:55:59,279 DEBUG [Thread-3043] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:59,280 DEBUG [Thread-3045] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x653bb8fc to 127.0.0.1:49755 2023-04-05 18:55:59,280 DEBUG [Thread-3045] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:59,283 DEBUG [Thread-3041] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x211582f2 to 127.0.0.1:49755 2023-04-05 18:55:59,283 DEBUG [Thread-3041] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:59,285 DEBUG [Thread-3039] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f5363d8 to 127.0.0.1:49755 2023-04-05 18:55:59,285 DEBUG [Thread-3039] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:55:59,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:59,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42008 deadline: 1680721019342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:59,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42080 deadline: 1680721019344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:59,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42032 deadline: 1680721019345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:59,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1680721019349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:55:59,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42020 deadline: 1680721019510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,676 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/904baf0a892a4186b94e6962916750af 2023-04-05 18:55:59,679 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/bf85bceed9cf408b81864445cdcf24b2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/bf85bceed9cf408b81864445cdcf24b2 2023-04-05 18:55:59,684 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/bf85bceed9cf408b81864445cdcf24b2, entries=150, sequenceid=574, filesize=11.9 K 2023-04-05 18:55:59,684 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/566d367afc27451e9453c077cbbe0c87 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/566d367afc27451e9453c077cbbe0c87 2023-04-05 18:55:59,687 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/566d367afc27451e9453c077cbbe0c87, entries=150, sequenceid=574, filesize=11.9 K 2023-04-05 18:55:59,688 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/904baf0a892a4186b94e6962916750af as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/904baf0a892a4186b94e6962916750af 2023-04-05 18:55:59,690 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/904baf0a892a4186b94e6962916750af, entries=150, sequenceid=574, filesize=11.9 K 2023-04-05 18:55:59,691 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 6e819dddd14a04b6eb4e71fe4e02a010 in 1247ms, sequenceid=574, compaction requested=true 2023-04-05 18:55:59,691 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:55:59,691 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:55:59,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:55:59,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:55:59,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:55:59,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:55:59,693 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,693 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:59,693 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:59,693 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@592b1f20[Count = 0] remaining members to acquire global barrier 2023-04-05 18:55:59,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:55:59,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:59,695 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:59,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:59,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:55:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:55:59,695 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,695 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:55:59,697 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:55:59,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:59,697 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:55:59,699 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:55:59,699 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:55:59,699 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:55:59,699 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:55:59,699 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:55:59,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:59,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:59,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:55:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:55:59,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:59,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:59,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:55:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:55:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:55:59,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:59,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:55:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:55:59,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:55:59,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:55:59,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:55:59,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:55:59,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:55:59,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:55:59,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:59,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:55:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:55:59,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:55:59,707 DEBUG [Thread-3038] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:55:59,707 DEBUG [Thread-3038] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:55:59,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:55:59,808 DEBUG [Thread-3038] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:55:59,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:00,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:56:00,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-05 18:56:00,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:56:00,016 DEBUG [Thread-3032] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06edd017 to 127.0.0.1:49755 2023-04-05 18:56:00,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:00,016 DEBUG [Thread-3032] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:00,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:56:00,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:00,017 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:56:00,017 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:00,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/31e638dee73d41739cb4e0014b6bc2d4 2023-04-05 18:56:00,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/65560458c229495aa03c9737e243f3dc 2023-04-05 18:56:01,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/b8d276ed16ac442c8d25eaf46ed01dec 2023-04-05 18:56:01,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/31e638dee73d41739cb4e0014b6bc2d4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/31e638dee73d41739cb4e0014b6bc2d4 2023-04-05 18:56:01,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/31e638dee73d41739cb4e0014b6bc2d4, entries=150, sequenceid=590, filesize=11.9 K 2023-04-05 18:56:01,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/65560458c229495aa03c9737e243f3dc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/65560458c229495aa03c9737e243f3dc 2023-04-05 18:56:01,252 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/65560458c229495aa03c9737e243f3dc, entries=150, sequenceid=590, filesize=11.9 K 2023-04-05 18:56:01,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/b8d276ed16ac442c8d25eaf46ed01dec as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b8d276ed16ac442c8d25eaf46ed01dec 2023-04-05 18:56:01,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b8d276ed16ac442c8d25eaf46ed01dec, entries=150, sequenceid=590, filesize=11.9 K 2023-04-05 18:56:01,257 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 6e819dddd14a04b6eb4e71fe4e02a010 in 1241ms, sequenceid=590, compaction requested=true 2023-04-05 18:56:01,257 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:56:01,257 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:01,257 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:56:01,257 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:56:01,257 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:01,257 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:01,258 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54918 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:56:01,258 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50038 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:56:01,258 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/A is initiating minor compaction (all files) 2023-04-05 18:56:01,258 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/B is initiating minor compaction (all files) 2023-04-05 18:56:01,258 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/A in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:56:01,258 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/B in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:56:01,258 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/ee6c46cd511549a1b119a6829fc9326d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/7b401e7b19cf43d9b989c939cf76526c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/bf85bceed9cf408b81864445cdcf24b2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/31e638dee73d41739cb4e0014b6bc2d4] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=53.6 K 2023-04-05 18:56:01,258 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b17c79badb9c468c829b40b8bc60a14f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bb04e3bc0fa54e34865a86a914f33f92, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/566d367afc27451e9453c077cbbe0c87, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/65560458c229495aa03c9737e243f3dc] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=48.9 K 2023-04-05 18:56:01,259 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting ee6c46cd511549a1b119a6829fc9326d, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680720956498 2023-04-05 18:56:01,259 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting b17c79badb9c468c829b40b8bc60a14f, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680720956498 2023-04-05 18:56:01,259 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7b401e7b19cf43d9b989c939cf76526c, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680720957131 2023-04-05 18:56:01,259 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting bb04e3bc0fa54e34865a86a914f33f92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680720957168 2023-04-05 18:56:01,259 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting bf85bceed9cf408b81864445cdcf24b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680720957196 2023-04-05 18:56:01,259 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 566d367afc27451e9453c077cbbe0c87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680720957196 2023-04-05 18:56:01,259 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 31e638dee73d41739cb4e0014b6bc2d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680720958884 2023-04-05 18:56:01,260 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 65560458c229495aa03c9737e243f3dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680720958884 2023-04-05 18:56:01,267 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#A#compaction#515 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:01,269 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#B#compaction#516 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:01,284 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/fbfd13fb753c4ef6a3e1f9eb5752b4b4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/fbfd13fb753c4ef6a3e1f9eb5752b4b4 2023-04-05 18:56:01,285 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/e69c6ba50f204a00a9490727ea95883a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/e69c6ba50f204a00a9490727ea95883a 2023-04-05 18:56:01,288 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/A of 6e819dddd14a04b6eb4e71fe4e02a010 into fbfd13fb753c4ef6a3e1f9eb5752b4b4(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:01,289 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:56:01,289 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/A, priority=12, startTime=1680720961257; duration=0sec 2023-04-05 18:56:01,289 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/B of 6e819dddd14a04b6eb4e71fe4e02a010 into e69c6ba50f204a00a9490727ea95883a(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:01,289 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:01,289 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:56:01,289 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/B, priority=12, startTime=1680720961257; duration=0sec 2023-04-05 18:56:01,289 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:01,289 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:01,290 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:01,290 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 6e819dddd14a04b6eb4e71fe4e02a010/C is initiating minor compaction (all files) 2023-04-05 18:56:01,290 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6e819dddd14a04b6eb4e71fe4e02a010/C in TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:56:01,290 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/567aab9c3e594ed6b8c1a3120ad5e79d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/904baf0a892a4186b94e6962916750af, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b8d276ed16ac442c8d25eaf46ed01dec] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp, totalSize=37.0 K 2023-04-05 18:56:01,290 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 567aab9c3e594ed6b8c1a3120ad5e79d, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680720957168 2023-04-05 18:56:01,291 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 904baf0a892a4186b94e6962916750af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680720957196 2023-04-05 18:56:01,291 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting b8d276ed16ac442c8d25eaf46ed01dec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680720958884 2023-04-05 18:56:01,298 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6e819dddd14a04b6eb4e71fe4e02a010#C#compaction#517 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:01,354 DEBUG [Thread-3036] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x461f5f3f to 127.0.0.1:49755 2023-04-05 18:56:01,354 DEBUG [Thread-3036] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:01,354 DEBUG [Thread-3028] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:49755 2023-04-05 18:56:01,354 DEBUG [Thread-3028] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:01,358 DEBUG [Thread-3030] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:49755 2023-04-05 18:56:01,358 DEBUG [Thread-3030] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:01,360 DEBUG [Thread-3034] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fb168ce to 127.0.0.1:49755 2023-04-05 18:56:01,361 DEBUG [Thread-3034] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 107 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 85 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1832 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5496 rows 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1835 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5505 rows 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1835 2023-04-05 18:56:01,361 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5505 rows 2023-04-05 18:56:01,362 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1846 2023-04-05 18:56:01,362 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5538 rows 2023-04-05 18:56:01,362 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(402): scanned 1840 2023-04-05 18:56:01,362 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(403): verified 5520 rows 2023-04-05 18:56:01,362 INFO [Listener at localhost/43705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 18:56:01,362 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37705401 to 127.0.0.1:49755 2023-04-05 18:56:01,362 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:01,365 INFO [Listener at localhost/43705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 18:56:01,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 18:56:01,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:01,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 18:56:01,371 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720961371"}]},"ts":"1680720961371"} 2023-04-05 18:56:01,372 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 18:56:01,469 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 18:56:01,470 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6e819dddd14a04b6eb4e71fe4e02a010, UNASSIGN}] 2023-04-05 18:56:01,471 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6e819dddd14a04b6eb4e71fe4e02a010, UNASSIGN 2023-04-05 18:56:01,472 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=6e819dddd14a04b6eb4e71fe4e02a010, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:01,472 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720961472"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720961472"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720961472"}]},"ts":"1680720961472"} 2023-04-05 18:56:01,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 18:56:01,473 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:56:01,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:56:01,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6e819dddd14a04b6eb4e71fe4e02a010, disabling compactions & flushes 2023-04-05 18:56:01,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions to complete for region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:56:01,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 18:56:01,708 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/c763e531e64c49cfa436a3ef41301051 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/c763e531e64c49cfa436a3ef41301051 2023-04-05 18:56:01,712 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6e819dddd14a04b6eb4e71fe4e02a010/C of 6e819dddd14a04b6eb4e71fe4e02a010 into c763e531e64c49cfa436a3ef41301051(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:01,712 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:56:01,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:56:01,712 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010., storeName=6e819dddd14a04b6eb4e71fe4e02a010/C, priority=13, startTime=1680720961257; duration=0sec 2023-04-05 18:56:01,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:56:01,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. after waiting 0 ms 2023-04-05 18:56:01,712 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:01,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:56:01,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6e819dddd14a04b6eb4e71fe4e02a010 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-05 18:56:01,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=A 2023-04-05 18:56:01,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:01,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=B 2023-04-05 18:56:01,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:01,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6e819dddd14a04b6eb4e71fe4e02a010, store=C 2023-04-05 18:56:01,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:01,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 18:56:02,119 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=600 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/e887b19d1f8344af820b147c997a8385 2023-04-05 18:56:02,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 18:56:02,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=600 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/293fa563fbfc40a8845c3a28bde621db 2023-04-05 18:56:02,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=600 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/976e17483a9b489f8f8c55bb8da45f7b 2023-04-05 18:56:02,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/A/e887b19d1f8344af820b147c997a8385 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e887b19d1f8344af820b147c997a8385 2023-04-05 18:56:02,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e887b19d1f8344af820b147c997a8385, entries=100, sequenceid=600, filesize=9.5 K 2023-04-05 18:56:02,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/B/293fa563fbfc40a8845c3a28bde621db as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/293fa563fbfc40a8845c3a28bde621db 2023-04-05 18:56:02,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/293fa563fbfc40a8845c3a28bde621db, entries=100, sequenceid=600, filesize=9.5 K 2023-04-05 18:56:02,950 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/.tmp/C/976e17483a9b489f8f8c55bb8da45f7b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/976e17483a9b489f8f8c55bb8da45f7b 2023-04-05 18:56:02,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/976e17483a9b489f8f8c55bb8da45f7b, entries=100, sequenceid=600, filesize=9.5 K 2023-04-05 18:56:02,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 6e819dddd14a04b6eb4e71fe4e02a010 in 1242ms, sequenceid=600, compaction requested=false 2023-04-05 18:56:02,977 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2c3afd2982114da690d89f1669d2fbdb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/eabeb98b45d9420f89b07c12dcd311aa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/208dbed4bd8747808c5fc3ae9f3414f8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d5ac58e007d34f7aa0343e55f327f2ba, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/56b718db957f49199b9c128414f5ca0f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d9bf50cd79a24b3a9cc69dba8eb28d25, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e5144af0cf214255af351f67282b9ab1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d64e0552d26e4694b4cfdeb4d5242e39, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c915c48c5ac64c1b95ed705904cf7f2a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b06f330ce0214079b8bc8dae6901f609, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6307f23be8194600955aaf864f2b4a13, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/899ec81701d242c292be2d28420d1fa7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/198683004234478dbeee8b58b25dbe38, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c6ac4c6027644456805cca292267db08, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/22c1aa6c50f3411da1920c3ab31eeb4c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2d992e7af1f44453a935076c350bb731, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80ce5d4cf8745c39639cf733ce815c8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/1217370e976a4a74b7fa3e64d51c10a7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/43253dee81274475b2f6022821976793, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/5c1ba6bf7d014d2393949c62978443e4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6b4beb6436924d068fc3622b8844cb57, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/22f69b9ccc8942c8ae0cd6beccc99c18, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/3666baf3790f40b1abb224729cf13240, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/248c3c7fb452477cae0aabe95e6c5fb2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/9d9dbe5617084600acfd51d31034b234, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/07fb969d0072401997e9c6e976ba2d3b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/85a61d3baa174f8a9ce337a7ee4400d8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e6ca6fca4dcc4f728d8805ddb6381d3e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/51da5d328f2c49779bb588ef932d0e70, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/82679c5d12574f1893657c9f3234902f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/0e897b574f994af6bbd31a0749cacfa8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/aa5fe798fa524b1585be55f87cea5fc1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80754f71c654b8bbc53ecabe36f0867, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d72f9ab20e1e4cddb765f6e9bf876a77, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b86ade27a1704a929e4c21ba28092f0b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/05779f7bf7b94a6ebae226b060ac5ca0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/339fb210ea00482fbc973aeb04840024, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/67c0bc99cc8545758ca27d81202ebc0e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/ee6c46cd511549a1b119a6829fc9326d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/7b401e7b19cf43d9b989c939cf76526c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/bf85bceed9cf408b81864445cdcf24b2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/31e638dee73d41739cb4e0014b6bc2d4] to archive 2023-04-05 18:56:02,978 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:56:02,979 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2c3afd2982114da690d89f1669d2fbdb to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2c3afd2982114da690d89f1669d2fbdb 2023-04-05 18:56:02,981 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/eabeb98b45d9420f89b07c12dcd311aa to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/eabeb98b45d9420f89b07c12dcd311aa 2023-04-05 18:56:02,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/208dbed4bd8747808c5fc3ae9f3414f8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/208dbed4bd8747808c5fc3ae9f3414f8 2023-04-05 18:56:02,983 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d5ac58e007d34f7aa0343e55f327f2ba to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d5ac58e007d34f7aa0343e55f327f2ba 2023-04-05 18:56:02,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/56b718db957f49199b9c128414f5ca0f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/56b718db957f49199b9c128414f5ca0f 2023-04-05 18:56:02,985 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d9bf50cd79a24b3a9cc69dba8eb28d25 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d9bf50cd79a24b3a9cc69dba8eb28d25 2023-04-05 18:56:02,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e5144af0cf214255af351f67282b9ab1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e5144af0cf214255af351f67282b9ab1 2023-04-05 18:56:02,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d64e0552d26e4694b4cfdeb4d5242e39 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d64e0552d26e4694b4cfdeb4d5242e39 2023-04-05 18:56:02,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c915c48c5ac64c1b95ed705904cf7f2a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c915c48c5ac64c1b95ed705904cf7f2a 2023-04-05 18:56:02,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b06f330ce0214079b8bc8dae6901f609 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b06f330ce0214079b8bc8dae6901f609 2023-04-05 18:56:02,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6307f23be8194600955aaf864f2b4a13 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6307f23be8194600955aaf864f2b4a13 2023-04-05 18:56:02,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/899ec81701d242c292be2d28420d1fa7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/899ec81701d242c292be2d28420d1fa7 2023-04-05 18:56:02,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/198683004234478dbeee8b58b25dbe38 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/198683004234478dbeee8b58b25dbe38 2023-04-05 18:56:02,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c6ac4c6027644456805cca292267db08 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/c6ac4c6027644456805cca292267db08 2023-04-05 18:56:02,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/22c1aa6c50f3411da1920c3ab31eeb4c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/22c1aa6c50f3411da1920c3ab31eeb4c 2023-04-05 18:56:02,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2d992e7af1f44453a935076c350bb731 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/2d992e7af1f44453a935076c350bb731 2023-04-05 18:56:03,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80ce5d4cf8745c39639cf733ce815c8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80ce5d4cf8745c39639cf733ce815c8 2023-04-05 18:56:03,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/1217370e976a4a74b7fa3e64d51c10a7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/1217370e976a4a74b7fa3e64d51c10a7 2023-04-05 18:56:03,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/43253dee81274475b2f6022821976793 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/43253dee81274475b2f6022821976793 2023-04-05 18:56:03,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/5c1ba6bf7d014d2393949c62978443e4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/5c1ba6bf7d014d2393949c62978443e4 2023-04-05 18:56:03,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6b4beb6436924d068fc3622b8844cb57 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/6b4beb6436924d068fc3622b8844cb57 2023-04-05 18:56:03,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/22f69b9ccc8942c8ae0cd6beccc99c18 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/22f69b9ccc8942c8ae0cd6beccc99c18 2023-04-05 18:56:03,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/3666baf3790f40b1abb224729cf13240 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/3666baf3790f40b1abb224729cf13240 2023-04-05 18:56:03,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/248c3c7fb452477cae0aabe95e6c5fb2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/248c3c7fb452477cae0aabe95e6c5fb2 2023-04-05 18:56:03,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/9d9dbe5617084600acfd51d31034b234 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/9d9dbe5617084600acfd51d31034b234 2023-04-05 18:56:03,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/07fb969d0072401997e9c6e976ba2d3b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/07fb969d0072401997e9c6e976ba2d3b 2023-04-05 18:56:03,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/85a61d3baa174f8a9ce337a7ee4400d8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/85a61d3baa174f8a9ce337a7ee4400d8 2023-04-05 18:56:03,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e6ca6fca4dcc4f728d8805ddb6381d3e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e6ca6fca4dcc4f728d8805ddb6381d3e 2023-04-05 18:56:03,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/51da5d328f2c49779bb588ef932d0e70 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/51da5d328f2c49779bb588ef932d0e70 2023-04-05 18:56:03,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/82679c5d12574f1893657c9f3234902f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/82679c5d12574f1893657c9f3234902f 2023-04-05 18:56:03,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/0e897b574f994af6bbd31a0749cacfa8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/0e897b574f994af6bbd31a0749cacfa8 2023-04-05 18:56:03,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/aa5fe798fa524b1585be55f87cea5fc1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/aa5fe798fa524b1585be55f87cea5fc1 2023-04-05 18:56:03,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80754f71c654b8bbc53ecabe36f0867 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/a80754f71c654b8bbc53ecabe36f0867 2023-04-05 18:56:03,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d72f9ab20e1e4cddb765f6e9bf876a77 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/d72f9ab20e1e4cddb765f6e9bf876a77 2023-04-05 18:56:03,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b86ade27a1704a929e4c21ba28092f0b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/b86ade27a1704a929e4c21ba28092f0b 2023-04-05 18:56:03,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/05779f7bf7b94a6ebae226b060ac5ca0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/05779f7bf7b94a6ebae226b060ac5ca0 2023-04-05 18:56:03,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/339fb210ea00482fbc973aeb04840024 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/339fb210ea00482fbc973aeb04840024 2023-04-05 18:56:03,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/67c0bc99cc8545758ca27d81202ebc0e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/67c0bc99cc8545758ca27d81202ebc0e 2023-04-05 18:56:03,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/ee6c46cd511549a1b119a6829fc9326d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/ee6c46cd511549a1b119a6829fc9326d 2023-04-05 18:56:03,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/7b401e7b19cf43d9b989c939cf76526c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/7b401e7b19cf43d9b989c939cf76526c 2023-04-05 18:56:03,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/bf85bceed9cf408b81864445cdcf24b2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/bf85bceed9cf408b81864445cdcf24b2 2023-04-05 18:56:03,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/31e638dee73d41739cb4e0014b6bc2d4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/31e638dee73d41739cb4e0014b6bc2d4 2023-04-05 18:56:03,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f62e4bcb945c4336a3a595b1d41ffe2f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/e9700cfe63a847589fe12f1f990a9efa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/983ef6ecd07f40b6901c4eea05f35870, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b712f544f53943baa19796add81929ac, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/57f6a20fa19a400dbf533819520d9bc7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/00c3dd3121b14b5f8ae954d377dc8514, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/dc0c0f3bb9ed40bf9e03b570baf0588f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/294c81e8d85240339dc4f675607cabd6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/2d15eab9b82541138d7206e387aaddcf, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/01c85b5b24d748e59a135d8b2a90aebf, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bf5c18568ee846f897f20c3a5459e456, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/76a5dd5795b142269b3457b69cd8910e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/868c3038ea024ccc9f49fdda5e5b2500, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/50515664a0e1495da01aa7901ae81741, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ba6c637b1c8648719c715ad80aeb2f2b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed5070badd7f4e9cb5d41cdf812494b5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/9c4c40342e3b4d579b745d6e1509cc11, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/6544d0975dda45a7870276b428ad1edd, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/001ff8594dae47cf9a0fe22086491681, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/d61cf939fc5f4653bc089bddad76875c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/646ec1b855c042c1b1c70b7535e96224, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed29b398334e4d1984e52057ad724c0a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/04cfed0a136e4ac1b7a698e29cb54500, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/727bde48b9fd4a60b4efeed7160fa0f6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/de5b57497fbd49ed9c781e775caa9165, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b8cd06877a18472b8789894cb3764ff6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/1bcc2d521fb048a5b1378074a08e200a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/68944992e0b14fe6bd2871b839553035, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/4f1b29773c1845fdb22db0d31a4f78f8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/af5788f5535549ce974a8319ffdb0432, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/90e0413eff404d3a8199cd8255da62a6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/c19a90c94be047aa84a1fa3e2fdb125b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/222f8dda7157443e852584a4dcbfa7c9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f4ffd644f57446d29dcf48814a0c2515, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/154252475c874d52b756d0488414379d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/acfeea9deb3e45659d27c95a290b31e3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/13c77581ee6448f493927af48623163a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b17c79badb9c468c829b40b8bc60a14f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/77277dd98c834d15ba897fdd3d1c691b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bb04e3bc0fa54e34865a86a914f33f92, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/566d367afc27451e9453c077cbbe0c87, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/65560458c229495aa03c9737e243f3dc] to archive 2023-04-05 18:56:03,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:56:03,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f62e4bcb945c4336a3a595b1d41ffe2f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f62e4bcb945c4336a3a595b1d41ffe2f 2023-04-05 18:56:03,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/e9700cfe63a847589fe12f1f990a9efa to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/e9700cfe63a847589fe12f1f990a9efa 2023-04-05 18:56:03,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/983ef6ecd07f40b6901c4eea05f35870 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/983ef6ecd07f40b6901c4eea05f35870 2023-04-05 18:56:03,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b712f544f53943baa19796add81929ac to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b712f544f53943baa19796add81929ac 2023-04-05 18:56:03,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/57f6a20fa19a400dbf533819520d9bc7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/57f6a20fa19a400dbf533819520d9bc7 2023-04-05 18:56:03,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/00c3dd3121b14b5f8ae954d377dc8514 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/00c3dd3121b14b5f8ae954d377dc8514 2023-04-05 18:56:03,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/dc0c0f3bb9ed40bf9e03b570baf0588f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/dc0c0f3bb9ed40bf9e03b570baf0588f 2023-04-05 18:56:03,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/294c81e8d85240339dc4f675607cabd6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/294c81e8d85240339dc4f675607cabd6 2023-04-05 18:56:03,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/2d15eab9b82541138d7206e387aaddcf to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/2d15eab9b82541138d7206e387aaddcf 2023-04-05 18:56:03,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/01c85b5b24d748e59a135d8b2a90aebf to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/01c85b5b24d748e59a135d8b2a90aebf 2023-04-05 18:56:03,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bf5c18568ee846f897f20c3a5459e456 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bf5c18568ee846f897f20c3a5459e456 2023-04-05 18:56:03,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/76a5dd5795b142269b3457b69cd8910e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/76a5dd5795b142269b3457b69cd8910e 2023-04-05 18:56:03,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/868c3038ea024ccc9f49fdda5e5b2500 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/868c3038ea024ccc9f49fdda5e5b2500 2023-04-05 18:56:03,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/50515664a0e1495da01aa7901ae81741 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/50515664a0e1495da01aa7901ae81741 2023-04-05 18:56:03,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ba6c637b1c8648719c715ad80aeb2f2b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ba6c637b1c8648719c715ad80aeb2f2b 2023-04-05 18:56:03,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed5070badd7f4e9cb5d41cdf812494b5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed5070badd7f4e9cb5d41cdf812494b5 2023-04-05 18:56:03,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/9c4c40342e3b4d579b745d6e1509cc11 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/9c4c40342e3b4d579b745d6e1509cc11 2023-04-05 18:56:03,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/6544d0975dda45a7870276b428ad1edd to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/6544d0975dda45a7870276b428ad1edd 2023-04-05 18:56:03,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/001ff8594dae47cf9a0fe22086491681 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/001ff8594dae47cf9a0fe22086491681 2023-04-05 18:56:03,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/d61cf939fc5f4653bc089bddad76875c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/d61cf939fc5f4653bc089bddad76875c 2023-04-05 18:56:03,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/646ec1b855c042c1b1c70b7535e96224 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/646ec1b855c042c1b1c70b7535e96224 2023-04-05 18:56:03,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed29b398334e4d1984e52057ad724c0a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/ed29b398334e4d1984e52057ad724c0a 2023-04-05 18:56:03,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/04cfed0a136e4ac1b7a698e29cb54500 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/04cfed0a136e4ac1b7a698e29cb54500 2023-04-05 18:56:03,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/727bde48b9fd4a60b4efeed7160fa0f6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/727bde48b9fd4a60b4efeed7160fa0f6 2023-04-05 18:56:03,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/de5b57497fbd49ed9c781e775caa9165 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/de5b57497fbd49ed9c781e775caa9165 2023-04-05 18:56:03,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b8cd06877a18472b8789894cb3764ff6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b8cd06877a18472b8789894cb3764ff6 2023-04-05 18:56:03,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/1bcc2d521fb048a5b1378074a08e200a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/1bcc2d521fb048a5b1378074a08e200a 2023-04-05 18:56:03,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/68944992e0b14fe6bd2871b839553035 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/68944992e0b14fe6bd2871b839553035 2023-04-05 18:56:03,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/4f1b29773c1845fdb22db0d31a4f78f8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/4f1b29773c1845fdb22db0d31a4f78f8 2023-04-05 18:56:03,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/af5788f5535549ce974a8319ffdb0432 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/af5788f5535549ce974a8319ffdb0432 2023-04-05 18:56:03,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/90e0413eff404d3a8199cd8255da62a6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/90e0413eff404d3a8199cd8255da62a6 2023-04-05 18:56:03,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/c19a90c94be047aa84a1fa3e2fdb125b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/c19a90c94be047aa84a1fa3e2fdb125b 2023-04-05 18:56:03,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/222f8dda7157443e852584a4dcbfa7c9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/222f8dda7157443e852584a4dcbfa7c9 2023-04-05 18:56:03,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f4ffd644f57446d29dcf48814a0c2515 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/f4ffd644f57446d29dcf48814a0c2515 2023-04-05 18:56:03,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/154252475c874d52b756d0488414379d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/154252475c874d52b756d0488414379d 2023-04-05 18:56:03,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/acfeea9deb3e45659d27c95a290b31e3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/acfeea9deb3e45659d27c95a290b31e3 2023-04-05 18:56:03,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/13c77581ee6448f493927af48623163a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/13c77581ee6448f493927af48623163a 2023-04-05 18:56:03,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b17c79badb9c468c829b40b8bc60a14f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/b17c79badb9c468c829b40b8bc60a14f 2023-04-05 18:56:03,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/77277dd98c834d15ba897fdd3d1c691b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/77277dd98c834d15ba897fdd3d1c691b 2023-04-05 18:56:03,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bb04e3bc0fa54e34865a86a914f33f92 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/bb04e3bc0fa54e34865a86a914f33f92 2023-04-05 18:56:03,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/566d367afc27451e9453c077cbbe0c87 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/566d367afc27451e9453c077cbbe0c87 2023-04-05 18:56:03,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/65560458c229495aa03c9737e243f3dc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/65560458c229495aa03c9737e243f3dc 2023-04-05 18:56:03,119 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4456fe4c1c8449a0b835b8d94b308f16, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0ff12166cf9b4f6a8ac47bf9f0ff8804, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/ab7cdb51e4964425b38d5792f8540da0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/928052bb087a435dafdd44c505dc0e32, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32d69cda5ca945c8a44fde188ebe7cd5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0819993c38ef497abcc238bb9d449fb9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d567c5a660af4049a7cb6c1b1918ad97, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1aa9a1a7e297421abbcc87231ea06108, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/bb542525c603428995729d320f72478a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/8b8d76af55314e9ab2b385b88098704c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d4c71c22b4dd459aa4a0d31b45bde312, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3b53d6bfb3194852b995061562a029b1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32268a80bbb344a7a47c66f7bd475583, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f575b194c80b46ff96d521b01321f4f3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e9d9935804bd4919ae327d6c3199ba20, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f7e7ca277afc4c4bb0440c626487f9df, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/7583b797fb504076a8dd2c50c8b81d09, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/43d71b759c2d46a59a6d56341512d310, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d8de6755e9ee4506b670e5b03f0020d6, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/18d052da8f8b4c9cb02358e478e7dc9a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/14af12d18df149089453fc6798e0b1e4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/51d06c4d4474488686b3068a9930ae45, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/07401a3d82fb47d583d5ec36ecf815a1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/eb4af94ef8594ded98a78ab4552f5bb8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/93b0df5adc734e82b84ad399c052b390, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3e93789a210a44ceb7962f38487f70d2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/67760f112f0640978b07f36b5a9c0ab7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4830da92bc6b437d805dc1a08f78fe6f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/00181c70dcb4454084f9c4e9f2174702, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b65fbec8c20647199804fca4280f4a49, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d737f9d471904e748411a9f341e0f81f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e6220b7500084c02a57df45d0ed5ab5b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e7e8306a61264d1f8ce00627b62eb572, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1cabe40a0e2a4959869a030ad9bad33f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/c82b697c0bcf4aa4b172297cb863c6bb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/648245ea3bb946a18d4a20e133572080, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/85f110b32e0143aa8ee37441bcc36a40, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/6e5b90ff8d9344faa8ab30b0b66f5144, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/567aab9c3e594ed6b8c1a3120ad5e79d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0fa768bb1ea94edb8033eb9ca29ba17a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/904baf0a892a4186b94e6962916750af, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b8d276ed16ac442c8d25eaf46ed01dec] to archive 2023-04-05 18:56:03,120 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:56:03,121 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4456fe4c1c8449a0b835b8d94b308f16 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4456fe4c1c8449a0b835b8d94b308f16 2023-04-05 18:56:03,122 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0ff12166cf9b4f6a8ac47bf9f0ff8804 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0ff12166cf9b4f6a8ac47bf9f0ff8804 2023-04-05 18:56:03,123 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/ab7cdb51e4964425b38d5792f8540da0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/ab7cdb51e4964425b38d5792f8540da0 2023-04-05 18:56:03,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/928052bb087a435dafdd44c505dc0e32 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/928052bb087a435dafdd44c505dc0e32 2023-04-05 18:56:03,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32d69cda5ca945c8a44fde188ebe7cd5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32d69cda5ca945c8a44fde188ebe7cd5 2023-04-05 18:56:03,126 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0819993c38ef497abcc238bb9d449fb9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0819993c38ef497abcc238bb9d449fb9 2023-04-05 18:56:03,127 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d567c5a660af4049a7cb6c1b1918ad97 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d567c5a660af4049a7cb6c1b1918ad97 2023-04-05 18:56:03,129 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1aa9a1a7e297421abbcc87231ea06108 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1aa9a1a7e297421abbcc87231ea06108 2023-04-05 18:56:03,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/bb542525c603428995729d320f72478a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/bb542525c603428995729d320f72478a 2023-04-05 18:56:03,131 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/8b8d76af55314e9ab2b385b88098704c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/8b8d76af55314e9ab2b385b88098704c 2023-04-05 18:56:03,132 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d4c71c22b4dd459aa4a0d31b45bde312 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d4c71c22b4dd459aa4a0d31b45bde312 2023-04-05 18:56:03,133 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3b53d6bfb3194852b995061562a029b1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3b53d6bfb3194852b995061562a029b1 2023-04-05 18:56:03,134 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32268a80bbb344a7a47c66f7bd475583 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/32268a80bbb344a7a47c66f7bd475583 2023-04-05 18:56:03,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f575b194c80b46ff96d521b01321f4f3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f575b194c80b46ff96d521b01321f4f3 2023-04-05 18:56:03,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e9d9935804bd4919ae327d6c3199ba20 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e9d9935804bd4919ae327d6c3199ba20 2023-04-05 18:56:03,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f7e7ca277afc4c4bb0440c626487f9df to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/f7e7ca277afc4c4bb0440c626487f9df 2023-04-05 18:56:03,137 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/7583b797fb504076a8dd2c50c8b81d09 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/7583b797fb504076a8dd2c50c8b81d09 2023-04-05 18:56:03,138 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/43d71b759c2d46a59a6d56341512d310 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/43d71b759c2d46a59a6d56341512d310 2023-04-05 18:56:03,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d8de6755e9ee4506b670e5b03f0020d6 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d8de6755e9ee4506b670e5b03f0020d6 2023-04-05 18:56:03,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/18d052da8f8b4c9cb02358e478e7dc9a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/18d052da8f8b4c9cb02358e478e7dc9a 2023-04-05 18:56:03,140 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/14af12d18df149089453fc6798e0b1e4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/14af12d18df149089453fc6798e0b1e4 2023-04-05 18:56:03,141 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/51d06c4d4474488686b3068a9930ae45 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/51d06c4d4474488686b3068a9930ae45 2023-04-05 18:56:03,142 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/07401a3d82fb47d583d5ec36ecf815a1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/07401a3d82fb47d583d5ec36ecf815a1 2023-04-05 18:56:03,142 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/eb4af94ef8594ded98a78ab4552f5bb8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/eb4af94ef8594ded98a78ab4552f5bb8 2023-04-05 18:56:03,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/93b0df5adc734e82b84ad399c052b390 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/93b0df5adc734e82b84ad399c052b390 2023-04-05 18:56:03,144 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3e93789a210a44ceb7962f38487f70d2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/3e93789a210a44ceb7962f38487f70d2 2023-04-05 18:56:03,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/67760f112f0640978b07f36b5a9c0ab7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/67760f112f0640978b07f36b5a9c0ab7 2023-04-05 18:56:03,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4830da92bc6b437d805dc1a08f78fe6f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/4830da92bc6b437d805dc1a08f78fe6f 2023-04-05 18:56:03,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/00181c70dcb4454084f9c4e9f2174702 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/00181c70dcb4454084f9c4e9f2174702 2023-04-05 18:56:03,148 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b65fbec8c20647199804fca4280f4a49 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b65fbec8c20647199804fca4280f4a49 2023-04-05 18:56:03,149 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d737f9d471904e748411a9f341e0f81f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/d737f9d471904e748411a9f341e0f81f 2023-04-05 18:56:03,150 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e6220b7500084c02a57df45d0ed5ab5b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e6220b7500084c02a57df45d0ed5ab5b 2023-04-05 18:56:03,150 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e7e8306a61264d1f8ce00627b62eb572 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/e7e8306a61264d1f8ce00627b62eb572 2023-04-05 18:56:03,151 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1cabe40a0e2a4959869a030ad9bad33f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/1cabe40a0e2a4959869a030ad9bad33f 2023-04-05 18:56:03,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/c82b697c0bcf4aa4b172297cb863c6bb to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/c82b697c0bcf4aa4b172297cb863c6bb 2023-04-05 18:56:03,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/648245ea3bb946a18d4a20e133572080 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/648245ea3bb946a18d4a20e133572080 2023-04-05 18:56:03,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/85f110b32e0143aa8ee37441bcc36a40 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/85f110b32e0143aa8ee37441bcc36a40 2023-04-05 18:56:03,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/6e5b90ff8d9344faa8ab30b0b66f5144 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/6e5b90ff8d9344faa8ab30b0b66f5144 2023-04-05 18:56:03,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/567aab9c3e594ed6b8c1a3120ad5e79d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/567aab9c3e594ed6b8c1a3120ad5e79d 2023-04-05 18:56:03,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0fa768bb1ea94edb8033eb9ca29ba17a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/0fa768bb1ea94edb8033eb9ca29ba17a 2023-04-05 18:56:03,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/904baf0a892a4186b94e6962916750af to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/904baf0a892a4186b94e6962916750af 2023-04-05 18:56:03,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b8d276ed16ac442c8d25eaf46ed01dec to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/b8d276ed16ac442c8d25eaf46ed01dec 2023-04-05 18:56:03,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/recovered.edits/603.seqid, newMaxSeqId=603, maxSeqId=1 2023-04-05 18:56:03,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010. 2023-04-05 18:56:03,163 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6e819dddd14a04b6eb4e71fe4e02a010: 2023-04-05 18:56:03,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:56:03,165 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=6e819dddd14a04b6eb4e71fe4e02a010, regionState=CLOSED 2023-04-05 18:56:03,165 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720963165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720963165"}]},"ts":"1680720963165"} 2023-04-05 18:56:03,167 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-05 18:56:03,167 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 6e819dddd14a04b6eb4e71fe4e02a010, server=jenkins-hbase4.apache.org,33489,1680720816595 in 1.6930 sec 2023-04-05 18:56:03,168 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-05 18:56:03,168 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6e819dddd14a04b6eb4e71fe4e02a010, UNASSIGN in 1.6970 sec 2023-04-05 18:56:03,168 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720963168"}]},"ts":"1680720963168"} 2023-04-05 18:56:03,169 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 18:56:03,171 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 18:56:03,172 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8050 sec 2023-04-05 18:56:03,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 18:56:03,476 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-05 18:56:03,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 18:56:03,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:03,479 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:03,480 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:03,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-05 18:56:03,486 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:56:03,488 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/recovered.edits] 2023-04-05 18:56:03,490 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e887b19d1f8344af820b147c997a8385 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/e887b19d1f8344af820b147c997a8385 2023-04-05 18:56:03,491 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/fbfd13fb753c4ef6a3e1f9eb5752b4b4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/A/fbfd13fb753c4ef6a3e1f9eb5752b4b4 2023-04-05 18:56:03,499 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/293fa563fbfc40a8845c3a28bde621db to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/293fa563fbfc40a8845c3a28bde621db 2023-04-05 18:56:03,501 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/e69c6ba50f204a00a9490727ea95883a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/B/e69c6ba50f204a00a9490727ea95883a 2023-04-05 18:56:03,503 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/976e17483a9b489f8f8c55bb8da45f7b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/976e17483a9b489f8f8c55bb8da45f7b 2023-04-05 18:56:03,505 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/c763e531e64c49cfa436a3ef41301051 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/C/c763e531e64c49cfa436a3ef41301051 2023-04-05 18:56:03,508 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/recovered.edits/603.seqid to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010/recovered.edits/603.seqid 2023-04-05 18:56:03,508 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/6e819dddd14a04b6eb4e71fe4e02a010 2023-04-05 18:56:03,508 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:56:03,511 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:03,512 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 18:56:03,513 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 18:56:03,514 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:03,514 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 18:56:03,514 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680720963514"}]},"ts":"9223372036854775807"} 2023-04-05 18:56:03,517 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 18:56:03,517 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6e819dddd14a04b6eb4e71fe4e02a010, NAME => 'TestAcidGuarantees,,1680720938094.6e819dddd14a04b6eb4e71fe4e02a010.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 18:56:03,517 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 18:56:03,517 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680720963517"}]},"ts":"9223372036854775807"} 2023-04-05 18:56:03,520 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 18:56:03,522 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:03,523 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 45 msec 2023-04-05 18:56:03,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-05 18:56:03,584 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-05 18:56:03,599 INFO [Listener at localhost/43705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=230 (was 227) - Thread LEAK? -, OpenFileDescriptor=583 (was 539) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=372 (was 373), ProcessCount=169 (was 169), AvailableMemoryMB=1920 (was 1178) - AvailableMemoryMB LEAK? - 2023-04-05 18:56:03,619 INFO [Listener at localhost/43705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=230, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=372, ProcessCount=169, AvailableMemoryMB=1913 2023-04-05 18:56:03,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 18:56:03,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 18:56:03,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:03,628 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 18:56:03,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-05 18:56:03,629 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 18:56:03,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 18:56:03,631 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:03,633 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686 empty. 2023-04-05 18:56:03,634 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:03,634 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:56:03,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 18:56:03,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 18:56:04,043 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 18:56:04,045 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7a7f1bea453fa17f7cd1e6eff9f4a686, NAME => 'TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp 2023-04-05 18:56:04,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:56:04,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7a7f1bea453fa17f7cd1e6eff9f4a686, disabling compactions & flushes 2023-04-05 18:56:04,052 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. after waiting 0 ms 2023-04-05 18:56:04,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,052 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:04,054 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 18:56:04,055 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720964054"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720964054"}]},"ts":"1680720964054"} 2023-04-05 18:56:04,055 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 18:56:04,056 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 18:56:04,056 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720964056"}]},"ts":"1680720964056"} 2023-04-05 18:56:04,057 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 18:56:04,059 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, ASSIGN}] 2023-04-05 18:56:04,061 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, ASSIGN 2023-04-05 18:56:04,061 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33489,1680720816595; forceNewPlan=false, retain=false 2023-04-05 18:56:04,213 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=7a7f1bea453fa17f7cd1e6eff9f4a686, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:04,213 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720964213"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720964213"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720964213"}]},"ts":"1680720964213"} 2023-04-05 18:56:04,214 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:56:04,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 18:56:04,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7a7f1bea453fa17f7cd1e6eff9f4a686, NAME => 'TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:56:04,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:56:04,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,370 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,371 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A 2023-04-05 18:56:04,371 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A 2023-04-05 18:56:04,371 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:56:04,371 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7a7f1bea453fa17f7cd1e6eff9f4a686 columnFamilyName A 2023-04-05 18:56:04,372 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(310): Store=7a7f1bea453fa17f7cd1e6eff9f4a686/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:56:04,372 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,373 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B 2023-04-05 18:56:04,373 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B 2023-04-05 18:56:04,373 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:56:04,373 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7a7f1bea453fa17f7cd1e6eff9f4a686 columnFamilyName B 2023-04-05 18:56:04,373 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(310): Store=7a7f1bea453fa17f7cd1e6eff9f4a686/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:56:04,374 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,374 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C 2023-04-05 18:56:04,374 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C 2023-04-05 18:56:04,375 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:56:04,375 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7a7f1bea453fa17f7cd1e6eff9f4a686 columnFamilyName C 2023-04-05 18:56:04,375 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(310): Store=7a7f1bea453fa17f7cd1e6eff9f4a686/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:56:04,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:56:04,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 18:56:04,380 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7a7f1bea453fa17f7cd1e6eff9f4a686; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59047054, jitterRate=-0.12013033032417297}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:56:04,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:04,380 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., pid=56, masterSystemTime=1680720964365 2023-04-05 18:56:04,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,382 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=7a7f1bea453fa17f7cd1e6eff9f4a686, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:04,382 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720964381"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680720964381"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680720964381"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680720964381"}]},"ts":"1680720964381"} 2023-04-05 18:56:04,383 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-05 18:56:04,383 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 in 168 msec 2023-04-05 18:56:04,384 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-05 18:56:04,384 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, ASSIGN in 324 msec 2023-04-05 18:56:04,385 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 18:56:04,385 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720964385"}]},"ts":"1680720964385"} 2023-04-05 18:56:04,386 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 18:56:04,387 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 18:56:04,388 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 763 msec 2023-04-05 18:56:04,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 18:56:04,735 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-05 18:56:04,735 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x6f06dd50 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:04,741 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59185ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:04,742 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:04,743 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56694, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:04,744 DEBUG [Listener at localhost/43705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 18:56:04,746 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 18:56:04,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 18:56:04,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 18:56:04,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:04,757 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 18:56:04,758 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 18:56:04,758 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 18:56:04,759 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-05 18:56:04,762 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, REOPEN/MOVE}] 2023-04-05 18:56:04,762 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, REOPEN/MOVE 2023-04-05 18:56:04,762 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7a7f1bea453fa17f7cd1e6eff9f4a686, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:04,763 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720964762"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720964762"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720964762"}]},"ts":"1680720964762"} 2023-04-05 18:56:04,763 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:56:04,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7a7f1bea453fa17f7cd1e6eff9f4a686, disabling compactions & flushes 2023-04-05 18:56:04,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. after waiting 0 ms 2023-04-05 18:56:04,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-05 18:56:04,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:04,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:04,920 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 7a7f1bea453fa17f7cd1e6eff9f4a686 to self. 2023-04-05 18:56:04,921 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:04,921 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7a7f1bea453fa17f7cd1e6eff9f4a686, regionState=CLOSED 2023-04-05 18:56:04,921 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720964921"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720964921"}]},"ts":"1680720964921"} 2023-04-05 18:56:04,923 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-05 18:56:04,923 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 in 159 msec 2023-04-05 18:56:04,924 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,33489,1680720816595; forceNewPlan=false, retain=true 2023-04-05 18:56:05,074 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7a7f1bea453fa17f7cd1e6eff9f4a686, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,074 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720965074"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720965074"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720965074"}]},"ts":"1680720965074"} 2023-04-05 18:56:05,076 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:56:05,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7a7f1bea453fa17f7cd1e6eff9f4a686, NAME => 'TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.', STARTKEY => '', ENDKEY => ''} 2023-04-05 18:56:05,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:05,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 18:56:05,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:05,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:05,231 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:05,232 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A 2023-04-05 18:56:05,232 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A 2023-04-05 18:56:05,232 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:56:05,233 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7a7f1bea453fa17f7cd1e6eff9f4a686 columnFamilyName A 2023-04-05 18:56:05,234 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(310): Store=7a7f1bea453fa17f7cd1e6eff9f4a686/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:56:05,234 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:05,234 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B 2023-04-05 18:56:05,234 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B 2023-04-05 18:56:05,235 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:56:05,235 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7a7f1bea453fa17f7cd1e6eff9f4a686 columnFamilyName B 2023-04-05 18:56:05,235 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(310): Store=7a7f1bea453fa17f7cd1e6eff9f4a686/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:56:05,235 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:05,236 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C 2023-04-05 18:56:05,236 DEBUG [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C 2023-04-05 18:56:05,236 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 18:56:05,236 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7a7f1bea453fa17f7cd1e6eff9f4a686 columnFamilyName C 2023-04-05 18:56:05,237 INFO [StoreOpener-7a7f1bea453fa17f7cd1e6eff9f4a686-1] regionserver.HStore(310): Store=7a7f1bea453fa17f7cd1e6eff9f4a686/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 18:56:05,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:05,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:05,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 18:56:05,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:05,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7a7f1bea453fa17f7cd1e6eff9f4a686; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70736974, jitterRate=0.05406305193901062}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 18:56:05,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:05,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., pid=61, masterSystemTime=1680720965227 2023-04-05 18:56:05,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,251 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7a7f1bea453fa17f7cd1e6eff9f4a686, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,252 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720965251"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680720965251"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680720965251"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680720965251"}]},"ts":"1680720965251"} 2023-04-05 18:56:05,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-05 18:56:05,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 in 177 msec 2023-04-05 18:56:05,255 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-05 18:56:05,255 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, REOPEN/MOVE in 493 msec 2023-04-05 18:56:05,257 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-05 18:56:05,257 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 496 msec 2023-04-05 18:56:05,258 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 509 msec 2023-04-05 18:56:05,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-05 18:56:05,259 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:05,274 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:05,274 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:05,278 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:05,278 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:05,282 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:05,282 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x06799455 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:05,292 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25baee3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:05,292 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:05,296 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:05,296 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x3b59bdcf to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:05,318 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a470a3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:05,318 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:05,338 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:05,338 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x32bf16cd to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:05,359 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dca23c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:05,359 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x28164190 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:05,376 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62d52321, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:05,377 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(139): Connect 0x308b3f39 to 127.0.0.1:49755 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 18:56:05,380 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a8bf279, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 18:56:05,390 DEBUG [hconnection-0x423d44fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:05,391 DEBUG [hconnection-0x6c0ccc18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:05,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:05,396 DEBUG [hconnection-0x27c09acb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:05,397 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56720, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:05,397 DEBUG [hconnection-0x7662078b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:05,397 DEBUG [hconnection-0x783f2707-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:05,397 DEBUG [hconnection-0xe62d7d1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:05,401 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56696, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:05,401 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56708, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:05,401 DEBUG [hconnection-0x1ed2ffef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:05,403 DEBUG [hconnection-0x352f6387-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:05,403 DEBUG [hconnection-0x5dd5167a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:05,405 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56730, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:05,408 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56746, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:05,409 DEBUG [hconnection-0x37d8c88-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 18:56:05,409 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56722, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:05,410 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:05,410 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:05,417 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56748, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:05,418 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56778, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 18:56:05,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:05,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:05,418 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:05,418 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:05,418 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:05,418 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:05,420 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,420 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,421 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:05,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:05,422 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,422 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:05,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:05,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 18:56:05,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:05,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:05,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:05,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:05,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:05,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:05,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:05,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:05,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:05,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:05,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:05,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:05,426 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,426 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:05,426 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:05,426 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,427 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:05,427 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:05,427 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,427 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:05,427 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:05,429 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,429 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,430 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fae3bdf[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:05,431 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:05,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,433 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:05,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:05,434 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:05,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,434 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:05,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:05,438 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:05,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:05,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:05,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:05,438 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:05,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:05,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721025440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721025441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721025442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:05,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721025443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:05,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:05,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:05,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:05,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:05,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:05,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,445 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:05,445 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:05,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:05,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:05,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721025444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:05,545 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:05,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:05,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721025545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721025545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721025545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721025546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721025546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:05,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:05,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:05,549 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:05,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:05,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:05,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:05,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:05,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:05,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:05,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:05,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:05,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:05,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:05,552 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:05,552 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:05,552 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:05,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:05,553 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:05,553 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,554 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:05,554 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:05,554 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:05,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,555 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,556 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,556 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@726f22d6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:05,557 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:05,557 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,558 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:05,558 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,558 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:05,561 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:05,562 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:05,562 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:05,564 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:05,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:05,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:05,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:05,564 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:05,566 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,566 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:05,566 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,566 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:05,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:05,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:05,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:05,575 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:05,576 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:05,576 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:05,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:05,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:05,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:05,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:05,676 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:05,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:05,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:05,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:05,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:05,679 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:05,679 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:05,679 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:05,679 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:05,680 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,680 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,682 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:05,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:05,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:05,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:05,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:05,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:05,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:05,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:05,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:05,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:05,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:05,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:05,685 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:05,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:05,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:05,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:05,686 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,686 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,687 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,687 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39170f4c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:05,688 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:05,689 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,689 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:05,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:05,691 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:05,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:05,694 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:05,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:05,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:05,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:05,694 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:05,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:05,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:05,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,705 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:05,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:05,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:05,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:05,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:05,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:05,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,709 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:05,709 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:05,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:05,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:05,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:05,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721025748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721025748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721025749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721025749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:05,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721025749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,809 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:05,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:05,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:05,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:05,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:05,812 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:05,812 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:05,812 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:05,812 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:05,812 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,813 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,814 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:05,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:05,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:05,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,814 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:05,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:05,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:05,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:05,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:05,815 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:05,815 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:05,815 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:05,816 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:05,816 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:05,816 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,816 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:05,816 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:05,816 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,816 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:05,816 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:05,816 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,816 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:05,816 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:05,818 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,818 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:05,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c4d551[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:05,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:05,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,823 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:05,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:05,824 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:05,824 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:05,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:05,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:05,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:05,827 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:05,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:05,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:05,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:05,827 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:05,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:05,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,831 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,833 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:05,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:05,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:05,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:05,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:05,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:05,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:05,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:05,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,834 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,834 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,834 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:05,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:05,834 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:05,834 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:05,841 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040505c198cf13ff48c0ab9dd425278ad2cd to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040505c198cf13ff48c0ab9dd425278ad2cd 2023-04-05 18:56:05,854 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/2535a52f1fea42c68a54e67bcde09a6f 2023-04-05 18:56:05,934 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:05,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:05,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:05,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:05,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:05,942 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:05,942 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:05,942 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:05,942 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:05,943 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,943 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,944 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:05,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:05,946 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,946 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,946 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:05,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:05,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:05,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:05,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:05,953 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,953 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:05,953 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:05,953 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:05,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:05,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:05,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:05,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:05,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,958 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:05,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:05,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@767a7155[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:05,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:05,960 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,961 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,961 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:05,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,961 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:05,961 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:05,962 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:05,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:05,963 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:05,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:05,966 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:05,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:05,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:05,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:05,966 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:05,969 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,969 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:05,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,969 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:05,970 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,971 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:05,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:05,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:05,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:05,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:05,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:05,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:05,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:05,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:05,973 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:05,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:05,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:05,973 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:05,973 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:05,973 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:06,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:06,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721026050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:06,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721026052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:06,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721026053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:06,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721026053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:06,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721026053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,074 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:06,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:06,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:06,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:06,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:06,077 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:06,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:06,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:06,078 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,078 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,079 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:06,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:06,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:06,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:06,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:06,081 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:06,081 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,081 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:06,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:06,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:06,082 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,082 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:06,082 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:06,082 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:06,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:06,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:06,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:06,087 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,087 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,089 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,089 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:06,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:06,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d0894de[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:06,091 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:06,091 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,092 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,092 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:06,092 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:06,092 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:06,092 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,092 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:06,093 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,093 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:06,093 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:06,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,093 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:06,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:06,097 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:06,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:06,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:06,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:06,097 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:06,098 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,098 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,099 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,100 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,101 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:06,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:06,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,110 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:06,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:06,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:06,111 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:06,111 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:06,211 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:06,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:06,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:06,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:06,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:06,215 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:06,215 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,215 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:06,215 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:06,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,217 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,217 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,217 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,217 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:06,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:06,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:06,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:06,218 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:06,218 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,218 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:06,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:06,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:06,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:06,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:06,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:06,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:06,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:06,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:06,231 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,231 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,231 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,231 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:06,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c64995c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:06,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:06,233 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,234 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,234 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:06,234 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:06,234 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:06,234 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,235 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:06,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:06,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:06,235 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:06,236 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:06,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:06,238 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:06,239 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:06,239 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:06,239 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:06,239 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:06,240 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,240 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:06,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,241 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:06,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:06,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,242 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,244 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:06,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:06,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:06,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,247 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:06,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,247 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:06,247 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:06,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,247 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:06,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:06,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:06,281 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/4de162e188b54af9a07c2a4736afaf00 2023-04-05 18:56:06,347 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:06,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:06,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:06,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:06,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:06,352 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:06,352 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,352 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:06,352 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:06,353 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,353 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,354 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,354 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,354 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,354 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:06,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:06,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:06,355 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:06,355 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,355 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:06,355 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:06,355 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:06,356 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,356 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:06,356 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:06,356 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:06,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:06,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:06,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:06,357 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,357 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,357 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,357 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:06,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6294a5dc[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:06,359 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:06,359 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,360 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,360 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:06,360 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:06,360 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:06,360 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,360 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:06,362 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:06,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:06,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:06,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:06,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:06,365 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:06,366 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:06,366 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:06,366 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:06,366 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:06,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:06,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:06,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:06,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,370 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,372 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:06,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:06,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:06,377 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:06,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,377 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:06,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:06,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:06,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,377 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,377 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:06,478 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:06,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:06,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:06,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:06,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:06,480 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:06,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:06,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:06,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,482 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,482 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,482 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,482 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:06,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:06,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:06,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:06,483 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:06,484 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,484 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:06,484 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:06,484 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:06,484 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,484 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:06,484 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:06,484 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,484 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:06,487 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,487 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,487 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,487 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:06,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:06,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c19002[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:06,488 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:06,488 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,491 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:06,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:06,492 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:06,492 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,492 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:06,493 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,493 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:06,493 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:06,493 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:06,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:06,497 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:06,497 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:06,497 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:06,497 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:06,497 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:06,501 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,501 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,501 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:06,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:06,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:06,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,503 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,504 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,506 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:06,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:06,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:06,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:06,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,506 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:06,506 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:06,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,507 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:06,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:06,507 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,507 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,508 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,508 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,508 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,508 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,508 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:06,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721026556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:06,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721026557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721026558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721026558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:06,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721026559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,607 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:06,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:06,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:06,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:06,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:06,614 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:06,614 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,614 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:06,614 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:06,615 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,615 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,616 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,616 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,616 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,616 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:06,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:06,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:06,617 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:06,617 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,617 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:06,618 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:06,618 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:06,618 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,618 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:06,618 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:06,618 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,618 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:06,618 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:06,618 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:06,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,621 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:06,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e565240[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:06,623 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:06,624 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,625 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:06,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:06,625 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:06,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:06,626 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:06,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:06,630 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:06,630 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:06,630 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:06,630 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:06,630 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:06,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:06,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,632 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:06,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:06,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,632 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,634 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:06,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:06,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:06,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:06,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:06,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:06,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:06,638 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:06,639 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:06,639 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:06,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/bdd2ac88b5a34b07bb636c121cccd024 2023-04-05 18:56:06,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/2535a52f1fea42c68a54e67bcde09a6f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/2535a52f1fea42c68a54e67bcde09a6f 2023-04-05 18:56:06,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/2535a52f1fea42c68a54e67bcde09a6f, entries=150, sequenceid=17, filesize=25.1 K 2023-04-05 18:56:06,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/4de162e188b54af9a07c2a4736afaf00 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4de162e188b54af9a07c2a4736afaf00 2023-04-05 18:56:06,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4de162e188b54af9a07c2a4736afaf00, entries=150, sequenceid=17, filesize=11.6 K 2023-04-05 18:56:06,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/bdd2ac88b5a34b07bb636c121cccd024 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/bdd2ac88b5a34b07bb636c121cccd024 2023-04-05 18:56:06,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/bdd2ac88b5a34b07bb636c121cccd024, entries=150, sequenceid=17, filesize=11.6 K 2023-04-05 18:56:06,721 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 1297ms, sequenceid=17, compaction requested=false 2023-04-05 18:56:06,722 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:06,739 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:06,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:06,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:06,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:06,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:06,743 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:06,743 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,743 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:06,743 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:06,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,744 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,749 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,750 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:06,750 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:06,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:06,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:06,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:06,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:06,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:06,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:06,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:06,760 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:06,760 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:06,760 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 18:56:06,761 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:06,761 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:06,761 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:06,761 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:06,761 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:06,761 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:07,174 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304059744534d5569413b9f72b0473813590b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059744534d5569413b9f72b0473813590b 2023-04-05 18:56:07,184 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/eb75e8d442a644a7895b5f9a29475149 2023-04-05 18:56:07,214 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/d68de7fc83aa45e886a7afaef7245545 2023-04-05 18:56:07,243 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/97c3bf19cc7d4dec9397ca7dca1f2d6f 2023-04-05 18:56:07,248 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/eb75e8d442a644a7895b5f9a29475149 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/eb75e8d442a644a7895b5f9a29475149 2023-04-05 18:56:07,252 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/eb75e8d442a644a7895b5f9a29475149, entries=150, sequenceid=36, filesize=25.1 K 2023-04-05 18:56:07,253 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/d68de7fc83aa45e886a7afaef7245545 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d68de7fc83aa45e886a7afaef7245545 2023-04-05 18:56:07,256 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d68de7fc83aa45e886a7afaef7245545, entries=150, sequenceid=36, filesize=11.6 K 2023-04-05 18:56:07,257 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/97c3bf19cc7d4dec9397ca7dca1f2d6f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/97c3bf19cc7d4dec9397ca7dca1f2d6f 2023-04-05 18:56:07,260 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/97c3bf19cc7d4dec9397ca7dca1f2d6f, entries=150, sequenceid=36, filesize=11.6 K 2023-04-05 18:56:07,261 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 501ms, sequenceid=36, compaction requested=false 2023-04-05 18:56:07,261 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:07,261 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:07,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:07,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:07,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:07,263 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26a08aaf[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:07,266 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:07,266 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,267 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,267 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:07,267 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:07,267 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:07,268 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,268 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:07,269 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:07,269 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:07,269 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:07,269 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:07,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:07,272 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:07,272 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:07,272 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:07,272 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:07,272 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:07,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:07,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:07,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:07,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,277 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:07,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:07,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:07,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:07,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:07,279 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:07,279 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:07,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:07,279 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:07,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:07,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:07,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:07,379 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:07,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:07,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:07,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:07,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:07,383 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:07,383 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:07,383 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:07,383 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:07,384 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,384 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,385 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:07,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:07,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:07,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:07,386 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:07,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:07,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:07,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:07,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:07,387 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,387 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:07,387 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:07,387 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:07,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:07,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,387 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:07,388 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:07,389 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,389 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,389 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,389 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:07,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@615d71e2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:07,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:07,393 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,395 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,395 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:07,395 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:07,395 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:07,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,396 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:07,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:07,397 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:07,397 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:07,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:07,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:07,405 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:07,405 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:07,405 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:07,405 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:07,405 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:07,410 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,410 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,410 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:07,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:07,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:07,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,411 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,413 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:07,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:07,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:07,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:07,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:07,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:07,417 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:07,417 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:07,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,418 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,418 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,417 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:07,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:07,517 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:07,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:07,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:07,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:07,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:07,521 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:07,521 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:07,521 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:07,521 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:07,522 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,522 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,523 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,523 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:07,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:07,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:07,523 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,523 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:07,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:07,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:07,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:07,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:07,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:07,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:07,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:07,528 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:07,528 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,528 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:07,529 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:07,529 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:07,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:07,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:07,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:07,530 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,530 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,531 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,531 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:07,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:07,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a57bc4b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:07,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:07,533 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,534 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:07,534 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,534 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:07,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:07,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:07,537 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:07,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:07,541 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:07,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:07,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:07,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:07,541 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:07,542 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,542 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,542 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:07,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,543 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,544 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,547 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:07,547 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:07,547 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,547 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,547 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:07,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:07,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:07,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:07,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:07,547 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:07,547 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:07,547 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,547 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,547 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,548 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:07,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:07,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:56:07,605 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:07,605 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:07,605 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:07,605 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:07,605 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:07,605 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:07,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720967619, max timestamp=1680720967632Num uniques 859; 2023-04-05 18:56:07,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:56:07,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720967619, max timestamp=1680720967632Num uniques 859; 2023-04-05 18:56:07,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:56:07,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720967619, max timestamp=1680720967632Num uniques 859; 2023-04-05 18:56:07,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:56:07,648 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:07,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:07,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:07,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:07,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:07,672 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:07,672 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:07,672 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:07,672 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:07,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721027665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721027665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721027666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721027666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,677 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:07,677 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:07,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721027672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:07,677 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,677 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:07,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:07,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:07,704 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,704 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:07,704 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405e8a9707d09f84a7d9434cfc1144df671 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e8a9707d09f84a7d9434cfc1144df671 2023-04-05 18:56:07,704 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:07,704 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:07,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,707 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:07,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@665f2f13[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:07,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:07,709 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:07,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:07,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:07,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:07,712 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:07,712 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:07,712 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,712 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:07,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:07,714 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:07,714 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:07,714 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:07,714 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:07,714 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:07,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,718 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,720 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:07,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:07,722 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:07,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:07,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,722 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:07,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:07,722 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/b869e2a0bbc0444fb698c522f483b9c8 2023-04-05 18:56:07,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:07,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:07,723 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:07,723 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:07,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:07,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:07,732 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/ee2da8ae8b6a4c37898b41a3444b5db4 2023-04-05 18:56:07,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/655bcaa5335640d0bf2ebcdc4c7f803f 2023-04-05 18:56:07,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/b869e2a0bbc0444fb698c522f483b9c8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b869e2a0bbc0444fb698c522f483b9c8 2023-04-05 18:56:07,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b869e2a0bbc0444fb698c522f483b9c8, entries=150, sequenceid=49, filesize=25.1 K 2023-04-05 18:56:07,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/ee2da8ae8b6a4c37898b41a3444b5db4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ee2da8ae8b6a4c37898b41a3444b5db4 2023-04-05 18:56:07,773 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ee2da8ae8b6a4c37898b41a3444b5db4, entries=150, sequenceid=49, filesize=11.6 K 2023-04-05 18:56:07,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/655bcaa5335640d0bf2ebcdc4c7f803f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/655bcaa5335640d0bf2ebcdc4c7f803f 2023-04-05 18:56:07,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/655bcaa5335640d0bf2ebcdc4c7f803f, entries=150, sequenceid=49, filesize=11.6 K 2023-04-05 18:56:07,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 176ms, sequenceid=49, compaction requested=true 2023-04-05 18:56:07,779 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:07,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:07,780 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:07,780 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:07,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:07,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:07,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:07,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-04-05 18:56:07,784 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:07,784 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:07,784 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:07,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:07,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:07,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:07,785 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:07,785 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/A is initiating minor compaction (all files) 2023-04-05 18:56:07,785 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/A in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,785 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/2535a52f1fea42c68a54e67bcde09a6f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/eb75e8d442a644a7895b5f9a29475149, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b869e2a0bbc0444fb698c522f483b9c8] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=75.4 K 2023-04-05 18:56:07,786 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:07,786 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:07,786 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,786 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4de162e188b54af9a07c2a4736afaf00, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d68de7fc83aa45e886a7afaef7245545, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ee2da8ae8b6a4c37898b41a3444b5db4] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=34.8 K 2023-04-05 18:56:07,786 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4de162e188b54af9a07c2a4736afaf00, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680720965418 2023-04-05 18:56:07,787 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting d68de7fc83aa45e886a7afaef7245545, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680720965435 2023-04-05 18:56:07,787 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ee2da8ae8b6a4c37898b41a3444b5db4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720967574 2023-04-05 18:56:07,788 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 2535a52f1fea42c68a54e67bcde09a6f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680720965418 2023-04-05 18:56:07,788 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting eb75e8d442a644a7895b5f9a29475149, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680720965435 2023-04-05 18:56:07,789 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting b869e2a0bbc0444fb698c522f483b9c8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720967574 2023-04-05 18:56:07,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721027792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,799 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#531 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:07,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721027797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721027797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721027797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721027798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,823 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:07,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:07,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:07,861 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#A#compaction#532 average throughput is 0.32 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:07,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:07,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:07,864 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:07,864 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:07,864 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:07,864 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:07,865 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,865 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,867 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:07,867 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:07,867 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,867 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:07,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:07,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:07,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:07,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:07,888 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,888 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:07,888 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:07,888 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:07,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:07,888 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:07,896 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,897 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,897 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,897 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:07,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44883485[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:07,899 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:07,899 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,900 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721027899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:07,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:07,900 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:07,900 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,901 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:07,902 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:07,902 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:07,902 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:07,902 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:07,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:07,905 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:07,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:07,905 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:07,906 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:07,906 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:07,910 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,910 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:07,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,911 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721027908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721027908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,912 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721027909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:07,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721027909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:07,913 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:07,919 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:07,919 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:07,919 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:07,919 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:07,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:07,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:07,919 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:07,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:07,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:07,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:07,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:07,920 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:07,920 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:07,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:07,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:07,926 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/98404499ba7a4aefa6fa05eb44cb1d59 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/98404499ba7a4aefa6fa05eb44cb1d59 2023-04-05 18:56:07,930 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/A of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 98404499ba7a4aefa6fa05eb44cb1d59(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:07,930 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:07,930 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/A, priority=13, startTime=1680720967779; duration=0sec 2023-04-05 18:56:07,930 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:07,930 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:07,931 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:07,931 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction (all files) 2023-04-05 18:56:07,931 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:07,932 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/bdd2ac88b5a34b07bb636c121cccd024, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/97c3bf19cc7d4dec9397ca7dca1f2d6f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/655bcaa5335640d0bf2ebcdc4c7f803f] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=34.8 K 2023-04-05 18:56:07,932 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting bdd2ac88b5a34b07bb636c121cccd024, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680720965418 2023-04-05 18:56:07,933 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 97c3bf19cc7d4dec9397ca7dca1f2d6f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680720965435 2023-04-05 18:56:07,933 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 655bcaa5335640d0bf2ebcdc4c7f803f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720967574 2023-04-05 18:56:07,940 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#533 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:08,020 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:08,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:08,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:08,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:08,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:08,024 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:08,024 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,024 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:08,024 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:08,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,025 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,026 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,026 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,027 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:08,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:08,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:08,032 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,032 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:08,032 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:08,032 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:08,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:08,034 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,034 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,034 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,034 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:08,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ed09f97[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:08,037 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:08,037 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,039 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,039 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:08,040 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:08,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:08,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:08,041 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,041 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:08,041 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,041 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:08,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:08,055 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:08,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:08,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:08,055 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:08,055 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:08,057 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,057 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,057 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,059 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,064 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,064 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,064 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:08,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:08,064 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,064 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,065 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,065 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,065 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,065 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,065 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,065 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:08,065 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:08,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721028102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721028113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721028113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721028113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721028114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,165 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:08,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:08,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:08,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:08,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:08,174 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:08,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:08,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:08,175 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,175 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,177 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,177 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,178 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,178 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:08,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:08,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:08,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:08,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:08,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:08,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:08,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:08,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:08,180 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,180 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:08,180 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:08,180 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:08,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:08,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:08,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:08,181 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,181 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,182 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,182 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:08,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f2c7e2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:08,183 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:08,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,185 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:08,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:08,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:08,185 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,185 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:08,186 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:08,186 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,186 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,186 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:08,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:08,190 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:08,190 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:08,190 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:08,190 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:08,190 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:08,192 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,194 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,196 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:08,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:08,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:08,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:08,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,197 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:08,197 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:08,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,254 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/afed465fa75449cbb4a13dfada3b85fc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/afed465fa75449cbb4a13dfada3b85fc 2023-04-05 18:56:08,280 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into afed465fa75449cbb4a13dfada3b85fc(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:08,280 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:08,280 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=13, startTime=1680720967780; duration=0sec 2023-04-05 18:56:08,280 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:08,281 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405ca6de1563ad94aeb8281eed2fa32f1d4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ca6de1563ad94aeb8281eed2fa32f1d4 2023-04-05 18:56:08,286 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/261071167e9f45c59df19d1ba7db9ecc 2023-04-05 18:56:08,297 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:08,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:08,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:08,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:08,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:08,300 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:08,301 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,301 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:08,301 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:08,301 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,301 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,302 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,302 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,302 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,302 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:08,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:08,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:08,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:08,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:08,304 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,304 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:08,304 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:08,304 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:08,305 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,306 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,306 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:08,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ca4c78[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:08,307 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:08,307 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,309 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,309 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:08,309 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:08,309 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:08,309 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,309 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:08,311 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,311 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:08,311 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,311 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:08,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:08,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:08,318 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:08,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:08,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:08,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:08,318 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:08,320 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,320 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,320 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,321 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,322 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:08,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:08,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,328 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,328 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,329 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:08,329 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:08,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:08,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,366 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/4dc89b2b93904cc4b534566e70807206 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/4dc89b2b93904cc4b534566e70807206 2023-04-05 18:56:08,370 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 4dc89b2b93904cc4b534566e70807206(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:08,370 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:08,370 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=13, startTime=1680720967780; duration=0sec 2023-04-05 18:56:08,371 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:08,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721028408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721028418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721028418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721028420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721028421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,429 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:08,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:08,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:08,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:08,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:08,432 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:08,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:08,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:08,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,435 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,436 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,436 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,436 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:08,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:08,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:08,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:08,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:08,440 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,440 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:08,440 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:08,440 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:08,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,442 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:08,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43636438[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:08,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:08,445 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,446 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:08,446 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:08,446 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:08,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,446 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:08,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:08,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,447 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,447 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:08,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:08,459 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:08,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:08,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:08,460 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:08,460 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:08,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,465 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:08,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:08,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,467 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,468 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,468 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:08,468 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:08,468 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,568 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:08,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:08,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:08,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:08,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:08,572 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:08,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:08,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:08,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,573 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,574 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,574 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:08,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:08,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:08,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:08,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,576 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:08,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,582 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@98567e4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:08,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:08,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,586 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:08,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:08,587 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,587 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:08,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:08,591 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:08,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:08,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:08,591 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:08,591 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:08,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,593 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,593 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,595 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,598 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:08,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:08,598 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:08,599 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:08,691 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 18:56:08,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/04810b45ddeb4dada8874fb0ce76cfab 2023-04-05 18:56:08,699 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:08,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:08,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:08,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:08,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:08,703 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:08,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:08,703 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:08,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:08,707 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:08,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:08,720 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:08,720 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,720 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:08,724 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:08,724 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:08,724 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,724 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:08,724 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:08,724 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,724 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:08,724 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:08,724 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:08,731 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,731 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,732 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,732 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:08,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46076ce1[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:08,736 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:08,736 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,738 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,738 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:08,738 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:08,738 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:08,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:08,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:08,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,740 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,740 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:08,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:08,752 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:08,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:08,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:08,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:08,752 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:08,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/962da7e23444443fb4c23652bda4ac2f 2023-04-05 18:56:08,754 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,754 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:08,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,755 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:08,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,755 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/261071167e9f45c59df19d1ba7db9ecc as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/261071167e9f45c59df19d1ba7db9ecc 2023-04-05 18:56:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,761 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,761 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:08,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,762 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:08,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:08,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:08,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:08,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,762 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:08,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:08,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:08,762 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:08,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:08,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/261071167e9f45c59df19d1ba7db9ecc, entries=200, sequenceid=85, filesize=31.9 K 2023-04-05 18:56:08,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/04810b45ddeb4dada8874fb0ce76cfab as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/04810b45ddeb4dada8874fb0ce76cfab 2023-04-05 18:56:08,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/04810b45ddeb4dada8874fb0ce76cfab, entries=150, sequenceid=85, filesize=11.6 K 2023-04-05 18:56:08,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/962da7e23444443fb4c23652bda4ac2f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/962da7e23444443fb4c23652bda4ac2f 2023-04-05 18:56:08,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/962da7e23444443fb4c23652bda4ac2f, entries=150, sequenceid=85, filesize=11.6 K 2023-04-05 18:56:08,778 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=46.96 KB/48090 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 994ms, sequenceid=85, compaction requested=false 2023-04-05 18:56:08,778 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:08,862 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:08,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:08,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:08,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:08,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:08,871 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:08,871 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,871 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:08,871 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:08,872 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,872 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,873 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:08,873 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,874 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,874 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:08,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:08,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:08,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:08,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:08,875 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:08,875 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:08,875 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:08,892 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:08,892 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:08,892 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:08,892 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:08,892 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:56:08,892 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:08,892 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:08,892 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:08,893 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:08,893 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:08,893 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:08,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:08,913 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:08,935 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304059c0b461658fa4cb5b27e4ef8b6236107 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059c0b461658fa4cb5b27e4ef8b6236107 2023-04-05 18:56:08,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680720968912, max timestamp=1680720968951Num uniques 859; 2023-04-05 18:56:08,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680720968912, max timestamp=1680720968951Num uniques 859; 2023-04-05 18:56:08,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-05 18:56:08,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680720968912, max timestamp=1680720968951Num uniques 859; 2023-04-05 18:56:08,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-05 18:56:08,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-05 18:56:08,957 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/0bf8315dce624d53847da0b98f100466 2023-04-05 18:56:08,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721028958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721028961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721028962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721028963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:08,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721028964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:08,974 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/d4defe034fb54b43b7edb10459e86bd8 2023-04-05 18:56:09,016 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/8ca0687f5af9466f8c21ef6673ecd621 2023-04-05 18:56:09,021 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/0bf8315dce624d53847da0b98f100466 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0bf8315dce624d53847da0b98f100466 2023-04-05 18:56:09,026 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0bf8315dce624d53847da0b98f100466, entries=100, sequenceid=98, filesize=18.4 K 2023-04-05 18:56:09,027 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/d4defe034fb54b43b7edb10459e86bd8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d4defe034fb54b43b7edb10459e86bd8 2023-04-05 18:56:09,031 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d4defe034fb54b43b7edb10459e86bd8, entries=100, sequenceid=98, filesize=9.3 K 2023-04-05 18:56:09,031 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/8ca0687f5af9466f8c21ef6673ecd621 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8ca0687f5af9466f8c21ef6673ecd621 2023-04-05 18:56:09,042 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8ca0687f5af9466f8c21ef6673ecd621, entries=100, sequenceid=98, filesize=9.3 K 2023-04-05 18:56:09,043 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=161.02 KB/164880 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 151ms, sequenceid=98, compaction requested=true 2023-04-05 18:56:09,043 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:09,043 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,043 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:09,043 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:09,043 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,044 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:09,044 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:09,047 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,047 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:09,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:09,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@523a51d5[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:09,050 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:09,050 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,052 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:09,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:09,052 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:09,052 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,053 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:09,053 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:09,053 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,053 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:09,054 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:09,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:09,057 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:09,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:09,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:09,057 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:09,057 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:09,061 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,061 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,061 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,061 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:09,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=167.72 KB heapSize=418.97 KB 2023-04-05 18:56:09,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:09,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:09,069 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:09,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:09,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:09,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:09,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:09,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:09,070 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:09,070 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:09,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721029071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721029072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721029073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721029073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721029074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,170 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:09,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:09,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:09,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:09,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:09,174 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:09,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:09,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:09,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,174 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,175 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,175 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,176 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:09,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:09,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:09,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:09,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:09,177 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,177 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:09,177 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:09,177 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:09,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721029176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,179 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,179 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:09,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a20bd0e[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:09,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:09,182 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721029179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721029180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721029180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721029180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,183 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,183 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:09,184 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:09,184 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:09,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,184 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:09,185 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:09,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,185 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:09,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:09,188 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:09,188 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:09,188 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:09,188 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:09,188 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:09,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,190 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,191 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,192 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:09,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:09,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,195 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:09,195 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:09,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,295 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:09,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:09,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:09,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:09,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:09,299 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:09,299 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,299 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:09,299 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:09,299 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,299 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,300 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,300 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,301 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,301 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:09,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:09,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:09,302 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,302 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:09,302 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:09,302 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:09,304 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,304 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,305 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:09,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@283841e9[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:09,306 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:09,306 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,308 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:09,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:09,308 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:09,309 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,309 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:09,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:09,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,310 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,310 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:09,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:09,312 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:09,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:09,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:09,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:09,312 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:09,314 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,314 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,314 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,318 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:09,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:09,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:09,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,318 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:09,318 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:09,318 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721029379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721029383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721029383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721029384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721029385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,418 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:09,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:09,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:09,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:09,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:09,421 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:09,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,421 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:09,422 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:09,422 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,422 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,424 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,424 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,424 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,424 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:09,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:09,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:09,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:09,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:09,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:09,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:09,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:09,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:09,426 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,426 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:09,426 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:09,426 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:09,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:09,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:09,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:09,427 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,427 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,428 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:09,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:09,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55a5506b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:09,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:09,429 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,433 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:09,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:09,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,433 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:09,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:09,435 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:09,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:09,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:09,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:09,438 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:09,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:09,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:09,438 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:09,438 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:09,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,439 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,440 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,440 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,441 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:09,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:09,444 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,445 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:09,445 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:09,445 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,484 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040505cbb268bb154027943a40babf1d3798 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040505cbb268bb154027943a40babf1d3798 2023-04-05 18:56:09,545 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:09,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:09,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:09,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:09,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:09,548 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:09,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:09,548 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:09,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,550 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,550 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,551 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:09,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:09,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:09,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:09,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:09,556 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:09,556 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,556 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:09,556 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:09,556 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:09,556 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,556 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:09,556 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:09,556 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:09,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:09,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:09,557 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:09,558 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,559 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,561 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,561 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:09,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:09,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559117f3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:09,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:09,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,563 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:09,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,564 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:09,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:09,566 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:09,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:09,569 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:09,569 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:09,569 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:09,569 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:09,569 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:09,571 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,571 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,571 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,572 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,574 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:09,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:09,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,578 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,578 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:09,579 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:09,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:09,679 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:09,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:09,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:09,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:09,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:09,682 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:09,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:09,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:09,682 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,683 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721029683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,688 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,688 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,688 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,688 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:09,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:09,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:09,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:09,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:09,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:09,689 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:09,690 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,690 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:09,690 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:09,690 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:09,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:09,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721029686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:09,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:09,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721029687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,691 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56720 deadline: 1680721029689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721029689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a560882[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:09,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:09,693 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,694 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,694 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:09,694 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:09,694 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:09,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:09,695 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:09,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:09,698 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:09,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:09,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:09,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:09,698 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:09,705 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,705 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,705 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,706 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,708 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,710 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:09,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:09,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,710 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,711 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:09,711 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:09,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:09,811 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:09,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:09,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:09,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:09,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:09,815 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:09,815 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,815 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:09,815 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:09,816 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,816 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,817 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,817 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,818 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,818 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:09,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:09,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:09,819 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,819 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:09,819 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:09,819 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:09,822 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,823 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:09,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@300b2d28[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:09,824 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:09,824 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,825 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:09,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:09,825 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:09,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,825 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:09,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:09,826 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:09,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:09,829 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:09,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:09,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:09,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:09,829 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:09,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:09,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:09,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,834 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,837 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:09,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:09,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:09,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:09,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:09,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:09,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:09,840 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:09,840 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:09,888 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=126, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/80aa58fd6e9945059b659ff33e04e0a8 2023-04-05 18:56:09,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/42ec2bc370ff411581baf87608048867 2023-04-05 18:56:09,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/f7c62e8bccad41709bd07263bac68d1b 2023-04-05 18:56:09,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/80aa58fd6e9945059b659ff33e04e0a8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/80aa58fd6e9945059b659ff33e04e0a8 2023-04-05 18:56:09,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/80aa58fd6e9945059b659ff33e04e0a8, entries=150, sequenceid=126, filesize=25.1 K 2023-04-05 18:56:09,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/42ec2bc370ff411581baf87608048867 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/42ec2bc370ff411581baf87608048867 2023-04-05 18:56:09,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/42ec2bc370ff411581baf87608048867, entries=150, sequenceid=126, filesize=11.6 K 2023-04-05 18:56:09,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/f7c62e8bccad41709bd07263bac68d1b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/f7c62e8bccad41709bd07263bac68d1b 2023-04-05 18:56:09,921 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/f7c62e8bccad41709bd07263bac68d1b, entries=150, sequenceid=126, filesize=11.6 K 2023-04-05 18:56:09,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~418.92 KB/428976, currentSize=33.54 KB/34350 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 853ms, sequenceid=126, compaction requested=true 2023-04-05 18:56:09,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:09,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:09,922 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:56:09,922 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:56:09,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:09,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:09,926 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:56:09,926 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:09,926 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,926 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/afed465fa75449cbb4a13dfada3b85fc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/04810b45ddeb4dada8874fb0ce76cfab, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d4defe034fb54b43b7edb10459e86bd8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/42ec2bc370ff411581baf87608048867] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=44.2 K 2023-04-05 18:56:09,926 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103079 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:56:09,926 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/A is initiating minor compaction (all files) 2023-04-05 18:56:09,927 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/A in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,927 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/98404499ba7a4aefa6fa05eb44cb1d59, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/261071167e9f45c59df19d1ba7db9ecc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0bf8315dce624d53847da0b98f100466, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/80aa58fd6e9945059b659ff33e04e0a8] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=100.7 K 2023-04-05 18:56:09,927 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting afed465fa75449cbb4a13dfada3b85fc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720967574 2023-04-05 18:56:09,927 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 04810b45ddeb4dada8874fb0ce76cfab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680720967664 2023-04-05 18:56:09,927 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 98404499ba7a4aefa6fa05eb44cb1d59, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720967574 2023-04-05 18:56:09,928 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting d4defe034fb54b43b7edb10459e86bd8, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680720967791 2023-04-05 18:56:09,928 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 261071167e9f45c59df19d1ba7db9ecc, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680720967648 2023-04-05 18:56:09,928 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 42ec2bc370ff411581baf87608048867, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680720968961 2023-04-05 18:56:09,928 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 0bf8315dce624d53847da0b98f100466, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680720967791 2023-04-05 18:56:09,929 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 80aa58fd6e9945059b659ff33e04e0a8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680720968961 2023-04-05 18:56:09,938 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#543 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:09,940 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:09,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:09,940 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#A#compaction#542 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:09,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:09,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:09,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:09,944 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:09,944 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,944 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:09,944 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:09,945 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,945 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,946 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,946 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:09,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:09,947 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:09,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:09,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:09,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:09,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:09,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:09,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:09,950 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,950 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:09,950 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 18:56:09,950 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:09,950 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:09,950 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:09,950 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:09,950 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:09,950 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:09,951 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/8a14479e3d1f477cbb4a3f6ce9bb842f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/8a14479e3d1f477cbb4a3f6ce9bb842f 2023-04-05 18:56:09,959 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 8a14479e3d1f477cbb4a3f6ce9bb842f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:09,959 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:09,959 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=12, startTime=1680720969922; duration=0sec 2023-04-05 18:56:09,959 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:09,959 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:56:09,962 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:56:09,962 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction (all files) 2023-04-05 18:56:09,962 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:09,962 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/4dc89b2b93904cc4b534566e70807206, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/962da7e23444443fb4c23652bda4ac2f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8ca0687f5af9466f8c21ef6673ecd621, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/f7c62e8bccad41709bd07263bac68d1b] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=44.2 K 2023-04-05 18:56:09,962 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4dc89b2b93904cc4b534566e70807206, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680720967574 2023-04-05 18:56:09,963 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 962da7e23444443fb4c23652bda4ac2f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680720967664 2023-04-05 18:56:09,963 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 8ca0687f5af9466f8c21ef6673ecd621, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680720967791 2023-04-05 18:56:09,963 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting f7c62e8bccad41709bd07263bac68d1b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680720968961 2023-04-05 18:56:09,970 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#544 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:09,986 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/58b5dd1df457451387784798d8737539 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/58b5dd1df457451387784798d8737539 2023-04-05 18:56:09,988 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/5b2cec50de40481ba327f2f43bc7ba35 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/5b2cec50de40481ba327f2f43bc7ba35 2023-04-05 18:56:09,990 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304051bbdf954addc48779a5821405848632c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051bbdf954addc48779a5821405848632c 2023-04-05 18:56:09,995 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/A of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 5b2cec50de40481ba327f2f43bc7ba35(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:09,995 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:09,995 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/A, priority=12, startTime=1680720969922; duration=0sec 2023-04-05 18:56:09,995 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:09,996 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 58b5dd1df457451387784798d8737539(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:09,996 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:09,996 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=12, startTime=1680720969922; duration=0sec 2023-04-05 18:56:09,996 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:09,998 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/56fed31f7090462c9290e1b8ba279c60 2023-04-05 18:56:10,011 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/4aeeea7e2e14470088905f3613844eca 2023-04-05 18:56:10,026 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/a7ff1a88ba49410285e45bed0fc2675d 2023-04-05 18:56:10,031 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/56fed31f7090462c9290e1b8ba279c60 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/56fed31f7090462c9290e1b8ba279c60 2023-04-05 18:56:10,035 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/56fed31f7090462c9290e1b8ba279c60, entries=150, sequenceid=134, filesize=25.3 K 2023-04-05 18:56:10,036 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/4aeeea7e2e14470088905f3613844eca as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4aeeea7e2e14470088905f3613844eca 2023-04-05 18:56:10,040 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4aeeea7e2e14470088905f3613844eca, entries=150, sequenceid=134, filesize=11.8 K 2023-04-05 18:56:10,041 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/a7ff1a88ba49410285e45bed0fc2675d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a7ff1a88ba49410285e45bed0fc2675d 2023-04-05 18:56:10,046 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a7ff1a88ba49410285e45bed0fc2675d, entries=150, sequenceid=134, filesize=11.8 K 2023-04-05 18:56:10,047 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 97ms, sequenceid=134, compaction requested=false 2023-04-05 18:56:10,047 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:10,047 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:10,050 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,051 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:10,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f52154a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:10,052 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:10,052 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,053 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,053 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:10,053 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:10,054 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:10,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,058 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:10,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:10,059 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,059 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:10,077 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:10,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:10,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:10,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:10,077 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:10,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,096 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:10,105 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:10,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:10,106 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:10,105 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:10,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:10,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:10,106 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,106 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,106 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,106 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:10,106 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:10,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:10,206 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:10,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:10,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:10,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:10,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:10,210 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:10,210 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:10,210 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:10,210 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:10,211 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,211 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,212 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,212 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:10,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:10,213 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,213 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:10,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:10,214 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:10,214 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:10,215 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-05 18:56:10,216 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:10,216 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:10,216 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:10,216 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:10,216 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:10,216 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:10,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:10,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:10,251 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405e416a1c1be2d4bf0a3214b06fb03229e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e416a1c1be2d4bf0a3214b06fb03229e 2023-04-05 18:56:10,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680720970252, max timestamp=1680720970259Num uniques 859; 2023-04-05 18:56:10,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 18:56:10,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720970256, max timestamp=1680720970259Num uniques 859; 2023-04-05 18:56:10,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:56:10,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720970256, max timestamp=1680720970259Num uniques 859; 2023-04-05 18:56:10,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:56:10,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720970263, max timestamp=1680720970296Num uniques 1068; 2023-04-05 18:56:10,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720970263, max timestamp=1680720970296Num uniques 1018; 2023-04-05 18:56:10,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680720970263, max timestamp=1680720970296Num uniques 1018; 2023-04-05 18:56:10,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-05 18:56:10,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 18:56:10,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 18:56:10,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680720970305, max timestamp=1680720970312Num uniques 1027; 2023-04-05 18:56:10,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680720970305, max timestamp=1680720970312Num uniques 1027; 2023-04-05 18:56:10,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12672, totalHeapSize=29048, min timestamp=1680720970305, max timestamp=1680720970312Num uniques 1027; 2023-04-05 18:56:10,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-05 18:56:10,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=12672, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-05 18:56:10,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-05 18:56:10,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721030346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680720970325, max timestamp=1680720970349Num uniques 1136; 2023-04-05 18:56:10,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721030347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-05 18:56:10,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680720970325, max timestamp=1680720970349Num uniques 1136; 2023-04-05 18:56:10,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680720970325, max timestamp=1680720970349Num uniques 1136; 2023-04-05 18:56:10,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-05 18:56:10,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-05 18:56:10,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680720970365, max timestamp=1680720970374Num uniques 1045; 2023-04-05 18:56:10,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=8920, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 18:56:10,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680720970365, max timestamp=1680720970374Num uniques 1045; 2023-04-05 18:56:10,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=8920, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 18:56:10,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680720970365, max timestamp=1680720970374Num uniques 1045; 2023-04-05 18:56:10,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=8920, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 18:56:10,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720970387, max timestamp=1680720970393Num uniques 1054; 2023-04-05 18:56:10,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 18:56:10,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720970387, max timestamp=1680720970393Num uniques 1054; 2023-04-05 18:56:10,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680720970387, max timestamp=1680720970393Num uniques 1054; 2023-04-05 18:56:10,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 18:56:10,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 18:56:10,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680720970436, max timestamp=1680720970441Num uniques 1013; 2023-04-05 18:56:10,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-05 18:56:10,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680720970436, max timestamp=1680720970441Num uniques 1013; 2023-04-05 18:56:10,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-05 18:56:10,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680720970436, max timestamp=1680720970441Num uniques 1013; 2023-04-05 18:56:10,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-05 18:56:10,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10152, totalHeapSize=23328, min timestamp=1680720970469, max timestamp=1680720970475Num uniques 1022; 2023-04-05 18:56:10,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=201, cellSize=9196, totalHeapSize=21144, min timestamp=1680720970469, max timestamp=1680720970475Num uniques 1022; 2023-04-05 18:56:10,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46792, new segment data size=9196, suffix heap size=134256,new segment heap size=21144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=201 2023-04-05 18:56:10,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10152, totalHeapSize=23328, min timestamp=1680720970469, max timestamp=1680720970475Num uniques 1022; 2023-04-05 18:56:10,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46792, new segment data size=10152, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-05 18:56:10,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46792, new segment data size=10152, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-05 18:56:10,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680720970490, max timestamp=1680720970495Num uniques 1081; 2023-04-05 18:56:10,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-05 18:56:10,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680720970490, max timestamp=1680720970495Num uniques 1081; 2023-04-05 18:56:10,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-05 18:56:10,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680720970490, max timestamp=1680720970495Num uniques 1060; 2023-04-05 18:56:10,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48550, new segment data size=10576, suffix heap size=138208,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1060, new segment cells count=231 2023-04-05 18:56:10,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721030510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680720970507, max timestamp=1680720970512Num uniques 1090; 2023-04-05 18:56:10,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-05 18:56:10,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680720970507, max timestamp=1680720970512Num uniques 1090; 2023-04-05 18:56:10,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-05 18:56:10,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680720970507, max timestamp=1680720970512Num uniques 1090; 2023-04-05 18:56:10,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-05 18:56:10,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721030533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9094, totalHeapSize=20936, min timestamp=1680720970525, max timestamp=1680720970536Num uniques 1099; 2023-04-05 18:56:10,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9107, totalHeapSize=20936, min timestamp=1680720970525, max timestamp=1680720970536Num uniques 1099; 2023-04-05 18:56:10,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50314, new segment data size=9094, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-05 18:56:10,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9105, totalHeapSize=20936, min timestamp=1680720970525, max timestamp=1680720970536Num uniques 1099; 2023-04-05 18:56:10,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50327, new segment data size=9107, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-05 18:56:10,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50325, new segment data size=9105, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-05 18:56:10,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680720970573, max timestamp=1680720970581Num uniques 1058; 2023-04-05 18:56:10,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-05 18:56:10,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680720970573, max timestamp=1680720970581Num uniques 1058; 2023-04-05 18:56:10,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-05 18:56:10,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680720970573, max timestamp=1680720970581Num uniques 1058; 2023-04-05 18:56:10,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-05 18:56:10,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680720970604, max timestamp=1680720970610Num uniques 1117; 2023-04-05 18:56:10,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-05 18:56:10,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680720970604, max timestamp=1680720970610Num uniques 1117; 2023-04-05 18:56:10,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-05 18:56:10,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680720970604, max timestamp=1680720970610Num uniques 1117; 2023-04-05 18:56:10,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-05 18:56:10,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680720970630, max timestamp=1680720970635Num uniques 1076; 2023-04-05 18:56:10,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680720970630, max timestamp=1680720970635Num uniques 1076; 2023-04-05 18:56:10,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680720970631, max timestamp=1680720970635Num uniques 1076; 2023-04-05 18:56:10,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-05 18:56:10,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-05 18:56:10,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-05 18:56:10,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721030657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680720970652, max timestamp=1680720970658Num uniques 1085; 2023-04-05 18:56:10,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721030657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-05 18:56:10,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680720970652, max timestamp=1680720970658Num uniques 1085; 2023-04-05 18:56:10,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-05 18:56:10,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680720970652, max timestamp=1680720970658Num uniques 1035; 2023-04-05 18:56:10,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-05 18:56:10,665 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/71505000faa54e0ba03f7d4773d23b3e 2023-04-05 18:56:10,676 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/b6dc9bfbb8bb4e60aad1dd56673be5f9 2023-04-05 18:56:10,685 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/e527745707ad406d87dd9937b15857ed 2023-04-05 18:56:10,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680720970675, max timestamp=1680720970684Num uniques 1044; 2023-04-05 18:56:10,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-05 18:56:10,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680720970675, max timestamp=1680720970684Num uniques 1044; 2023-04-05 18:56:10,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-05 18:56:10,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680720970675, max timestamp=1680720970684Num uniques 1044; 2023-04-05 18:56:10,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-05 18:56:10,689 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/71505000faa54e0ba03f7d4773d23b3e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71505000faa54e0ba03f7d4773d23b3e 2023-04-05 18:56:10,692 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71505000faa54e0ba03f7d4773d23b3e, entries=150, sequenceid=143, filesize=25.3 K 2023-04-05 18:56:10,693 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/b6dc9bfbb8bb4e60aad1dd56673be5f9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/b6dc9bfbb8bb4e60aad1dd56673be5f9 2023-04-05 18:56:10,696 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/b6dc9bfbb8bb4e60aad1dd56673be5f9, entries=100, sequenceid=143, filesize=9.4 K 2023-04-05 18:56:10,697 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/e527745707ad406d87dd9937b15857ed as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/e527745707ad406d87dd9937b15857ed 2023-04-05 18:56:10,699 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/e527745707ad406d87dd9937b15857ed, entries=100, sequenceid=143, filesize=9.4 K 2023-04-05 18:56:10,700 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=93.93 KB/96180 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 485ms, sequenceid=143, compaction requested=true 2023-04-05 18:56:10,700 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:10,700 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:10,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:10,700 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=107.34 KB heapSize=297.33 KB 2023-04-05 18:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,701 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:10,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:10,701 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:10,701 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:10,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:10,701 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:10,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:10,701 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,702 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:10,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1221addc[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:10,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:10,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,706 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:10,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:10,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:10,706 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,707 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:10,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:10,708 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:10,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,708 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:10,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:10,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:10,710 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:10,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:10,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:10,711 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:10,711 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:10,712 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405bf4cd97b04e94adcaad4d3585f4599e8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bf4cd97b04e94adcaad4d3585f4599e8 2023-04-05 18:56:10,712 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,712 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,712 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:10,713 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,716 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=432, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/c2640d28af664ffd8dee2b6aa6ade32e 2023-04-05 18:56:10,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:10,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:10,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:10,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,717 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:10,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:10,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:10,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:10,717 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:10,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:10,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,717 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:10,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:10,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721030719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721030720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721030732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721030762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,817 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:10,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:10,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:10,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:10,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:10,821 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:10,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:10,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:10,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:10,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,822 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,823 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:10,823 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:10,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:10,824 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,824 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:10,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:10,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:10,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:10,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:10,826 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:10,826 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:10,826 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:10,826 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:10,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721030825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,828 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,828 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:10,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:10,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7859b502[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:10,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:10,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,831 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:10,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,831 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,831 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:10,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:10,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721030829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:10,832 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:10,833 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:10,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,833 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:10,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:10,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:10,836 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:10,836 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:10,836 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:10,836 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:10,836 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:10,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721030835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,839 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:10,839 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,840 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:10,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:10,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:10,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,845 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:10,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:10,845 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:10,845 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:10,845 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:10,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:10,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:10,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721030862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,945 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:10,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:10,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:10,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:10,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:10,948 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:10,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:10,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:10,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:10,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:10,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:10,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:10,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:10,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:10,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:10,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:10,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:10,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:10,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:10,953 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:10,953 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:10,953 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:10,953 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:10,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:10,953 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:10,954 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,954 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:10,954 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:10,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,956 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:10,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:10,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d2a54c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:10,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:10,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,958 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,958 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:10,958 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:10,959 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:10,959 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,959 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:10,961 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,961 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:10,961 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,961 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:10,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:10,964 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:10,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:10,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:10,964 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:10,964 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:10,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:10,965 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:10,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:10,967 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,968 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:10,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:10,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,970 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:10,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:10,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:10,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:10,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:10,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:10,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:10,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:10,971 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:10,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:10,970 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:10,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:10,971 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:10,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:10,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:10,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721030968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721031028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721031032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721031039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,071 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:11,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:11,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:11,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:11,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:11,074 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:11,074 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:11,074 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:11,074 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:11,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,076 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:11,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:11,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:11,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:11,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:11,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:11,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:11,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:11,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:11,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:11,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:11,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:11,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:11,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:11,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:11,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:11,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:11,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,080 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:11,081 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,081 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:11,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:11,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e751a79[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:11,082 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:11,082 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,084 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:11,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:11,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:11,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:11,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,085 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:11,085 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:11,085 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:11,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:11,088 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:11,088 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:11,088 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:11,088 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:11,089 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:11,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:11,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:11,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:11,091 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:11,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,093 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:11,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:11,097 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:11,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:11,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:11,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:11,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,097 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:11,097 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:11,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:11,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:11,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/d3732ff3eb3941bc900c6a3ee5f642e4 2023-04-05 18:56:11,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/0aa9712a125c49c9a18c09a6a2edb8aa 2023-04-05 18:56:11,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/c2640d28af664ffd8dee2b6aa6ade32e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c2640d28af664ffd8dee2b6aa6ade32e 2023-04-05 18:56:11,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721031167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c2640d28af664ffd8dee2b6aa6ade32e, entries=300, sequenceid=432, filesize=46.1 K 2023-04-05 18:56:11,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/d3732ff3eb3941bc900c6a3ee5f642e4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3732ff3eb3941bc900c6a3ee5f642e4 2023-04-05 18:56:11,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3732ff3eb3941bc900c6a3ee5f642e4, entries=150, sequenceid=432, filesize=11.9 K 2023-04-05 18:56:11,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/0aa9712a125c49c9a18c09a6a2edb8aa as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/0aa9712a125c49c9a18c09a6a2edb8aa 2023-04-05 18:56:11,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/0aa9712a125c49c9a18c09a6a2edb8aa, entries=150, sequenceid=432, filesize=11.9 K 2023-04-05 18:56:11,179 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~297.28 KB/304416, currentSize=73.80 KB/75570 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 479ms, sequenceid=432, compaction requested=true 2023-04-05 18:56:11,179 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:11,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:11,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:56:11,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-05 18:56:11,179 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:56:11,179 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:56:11,181 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45990 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:56:11,181 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction (all files) 2023-04-05 18:56:11,181 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,181 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/58b5dd1df457451387784798d8737539, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a7ff1a88ba49410285e45bed0fc2675d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/e527745707ad406d87dd9937b15857ed, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/0aa9712a125c49c9a18c09a6a2edb8aa] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=44.9 K 2023-04-05 18:56:11,181 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 124944 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:56:11,182 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/A is initiating minor compaction (all files) 2023-04-05 18:56:11,182 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/A in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,182 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/5b2cec50de40481ba327f2f43bc7ba35, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/56fed31f7090462c9290e1b8ba279c60, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71505000faa54e0ba03f7d4773d23b3e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c2640d28af664ffd8dee2b6aa6ade32e] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=122.0 K 2023-04-05 18:56:11,182 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 58b5dd1df457451387784798d8737539, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680720968961 2023-04-05 18:56:11,182 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 5b2cec50de40481ba327f2f43bc7ba35, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680720968961 2023-04-05 18:56:11,182 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a7ff1a88ba49410285e45bed0fc2675d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680720969071 2023-04-05 18:56:11,182 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 56fed31f7090462c9290e1b8ba279c60, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680720969071 2023-04-05 18:56:11,183 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e527745707ad406d87dd9937b15857ed, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680720970213 2023-04-05 18:56:11,183 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 71505000faa54e0ba03f7d4773d23b3e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680720970200 2023-04-05 18:56:11,183 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 0aa9712a125c49c9a18c09a6a2edb8aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680720970698 2023-04-05 18:56:11,183 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c2640d28af664ffd8dee2b6aa6ade32e, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680720970690 2023-04-05 18:56:11,194 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#555 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:11,197 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:11,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:11,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:11,201 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#A#compaction#554 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:11,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:11,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:11,204 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:11,204 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:11,204 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:11,204 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:11,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,205 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:11,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:11,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:11,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:11,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:11,222 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:11,222 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:11,222 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:11,222 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:11,222 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:11,222 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,223 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:11,223 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 18:56:11,223 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:11,223 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:11,223 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:11,223 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:11,223 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:11,223 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:11,248 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/1eb2e363ca7e4777a973f4543fba68b5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/1eb2e363ca7e4777a973f4543fba68b5 2023-04-05 18:56:11,252 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 1eb2e363ca7e4777a973f4543fba68b5(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:11,252 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:11,252 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=12, startTime=1680720971179; duration=0sec 2023-04-05 18:56:11,252 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:11,252 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 18:56:11,253 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45990 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 18:56:11,254 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:11,254 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,254 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/8a14479e3d1f477cbb4a3f6ce9bb842f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4aeeea7e2e14470088905f3613844eca, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/b6dc9bfbb8bb4e60aad1dd56673be5f9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3732ff3eb3941bc900c6a3ee5f642e4] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=44.9 K 2023-04-05 18:56:11,254 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 8a14479e3d1f477cbb4a3f6ce9bb842f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680720968961 2023-04-05 18:56:11,255 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4aeeea7e2e14470088905f3613844eca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680720969071 2023-04-05 18:56:11,255 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting b6dc9bfbb8bb4e60aad1dd56673be5f9, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680720970213 2023-04-05 18:56:11,255 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting d3732ff3eb3941bc900c6a3ee5f642e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680720970698 2023-04-05 18:56:11,263 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#557 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:11,272 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405653dfc8045aa4df996102a982a4d39d7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405653dfc8045aa4df996102a982a4d39d7 2023-04-05 18:56:11,275 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/3597d99588d04002bf0cd432f90e6e55 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/3597d99588d04002bf0cd432f90e6e55 2023-04-05 18:56:11,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:11,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:11,278 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=446, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/e4ec620d4e6145558ae8512eace0b0de 2023-04-05 18:56:11,280 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 3597d99588d04002bf0cd432f90e6e55(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:11,280 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:11,280 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=12, startTime=1680720971179; duration=0sec 2023-04-05 18:56:11,280 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:11,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721031322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56696 deadline: 1680721031332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721031336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56722 deadline: 1680721031344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56730 deadline: 1680721031426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721031629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721031669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,695 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/1c531d22eca449b0b07d5722f5fbc889 2023-04-05 18:56:11,709 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/c0dd86d9676e4bb99ba03287fc65552c 2023-04-05 18:56:11,713 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/e4ec620d4e6145558ae8512eace0b0de as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/e4ec620d4e6145558ae8512eace0b0de 2023-04-05 18:56:11,717 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/e4ec620d4e6145558ae8512eace0b0de, entries=150, sequenceid=446, filesize=25.4 K 2023-04-05 18:56:11,718 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/1c531d22eca449b0b07d5722f5fbc889 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1c531d22eca449b0b07d5722f5fbc889 2023-04-05 18:56:11,722 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1c531d22eca449b0b07d5722f5fbc889, entries=150, sequenceid=446, filesize=11.9 K 2023-04-05 18:56:11,722 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/c0dd86d9676e4bb99ba03287fc65552c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c0dd86d9676e4bb99ba03287fc65552c 2023-04-05 18:56:11,727 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c0dd86d9676e4bb99ba03287fc65552c, entries=150, sequenceid=446, filesize=11.9 K 2023-04-05 18:56:11,727 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 504ms, sequenceid=446, compaction requested=false 2023-04-05 18:56:11,728 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:11,728 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:11,729 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,730 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,730 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:11,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:11,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a51126[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:11,731 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:11,731 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,732 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,732 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:11,732 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:11,732 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:11,733 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,733 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:11,734 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:11,735 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,735 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:11,735 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:11,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:11,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:11,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:11,738 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:11,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:11,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:11,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:11,738 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:11,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,739 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:11,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:11,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:11,740 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,741 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,746 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,746 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:11,746 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:11,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:11,746 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:11,746 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:11,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:11,746 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:11,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:11,747 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:11,747 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:11,746 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,747 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,747 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,747 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:11,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:11,838 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 18:56:11,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:11,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:11,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:11,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:11,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:11,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:11,847 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:11,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:11,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:11,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:11,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:11,855 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:11,855 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:11,855 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:11,855 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:11,855 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,856 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,857 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,857 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:11,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:11,857 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,857 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:11,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:11,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:11,859 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:11,859 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:11,859 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:11,859 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:11,859 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:11,859 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,860 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:11,860 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:11,861 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:11,861 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405a6ade5100bba432586b8e39a8417a146 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a6ade5100bba432586b8e39a8417a146 2023-04-05 18:56:11,863 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,864 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:11,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:11,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64615fb5[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:11,866 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:11,866 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,867 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=467, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/09efbb489451490eb0b08a53197bf224 2023-04-05 18:56:11,868 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:11,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:11,868 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:11,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721031863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721031864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721031864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,869 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:11,871 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,874 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:11,874 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:11,874 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:11,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:11,876 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:11,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:11,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:11,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:11,876 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:11,877 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:11,878 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:11,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,880 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,880 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:11,880 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/322a469e1a4f4a15b4b4269889028549 2023-04-05 18:56:11,884 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:11,884 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,884 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:11,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:11,885 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:11,885 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:11,884 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,884 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:11,885 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:11,885 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:11,885 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:11,885 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:11,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721031932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721031969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721031970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:11,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721031970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,985 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:11,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:11,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:11,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:11,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:11,989 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:11,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:11,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:11,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:11,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,989 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,991 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:11,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:11,991 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,992 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,992 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:11,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:11,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:11,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:11,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:11,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:11,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:11,993 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:11,993 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:11,993 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:11,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:11,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:11,994 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,994 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:11,994 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:11,994 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:11,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:11,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:11,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:11,994 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:11,996 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,996 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:11,996 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:11,996 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:11,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:11,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:11,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:11,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:11,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:11,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@601d7eb0[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:11,999 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:11,999 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,000 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:12,000 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,000 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:12,002 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:12,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:12,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:12,005 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:12,005 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:12,005 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:12,005 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:12,006 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:12,007 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,007 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,008 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,009 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,010 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,013 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:12,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:12,013 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:12,014 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:12,114 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:12,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:12,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:12,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:12,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:12,118 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:12,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:12,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:12,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,118 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,123 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:12,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:12,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:12,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:12,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:12,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:12,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:12,129 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:12,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d0618[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:12,132 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:12,132 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,134 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:12,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:12,135 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:12,135 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,135 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:12,136 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:12,136 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,136 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,136 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:12,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:12,139 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:12,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:12,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:12,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:12,139 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:12,140 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,140 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,140 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,142 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,143 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,148 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:12,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:12,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:12,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,148 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:12,148 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:12,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,149 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721032172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721032172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721032173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,248 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:12,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:12,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:12,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:12,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:12,252 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:12,252 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,252 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:12,252 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:12,252 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,252 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,253 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,253 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,254 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,254 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:12,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:12,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:12,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:12,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:12,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:12,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:12,255 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,255 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:12,255 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:12,255 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,255 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:12,256 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:12,256 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,256 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:12,256 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:12,260 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,260 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:12,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f881bd4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:12,262 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:12,262 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,263 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:12,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:12,263 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:12,263 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,263 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:12,264 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,265 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:12,265 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,265 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:12,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:12,267 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:12,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:12,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:12,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:12,267 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:12,268 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,268 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,269 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,269 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,270 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,274 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:12,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:12,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:12,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,274 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:12,274 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:12,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/3e8f4b7284c94a93b6deccafb4ec9cc7 2023-04-05 18:56:12,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/09efbb489451490eb0b08a53197bf224 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/09efbb489451490eb0b08a53197bf224 2023-04-05 18:56:12,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/09efbb489451490eb0b08a53197bf224, entries=200, sequenceid=467, filesize=32.3 K 2023-04-05 18:56:12,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/322a469e1a4f4a15b4b4269889028549 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/322a469e1a4f4a15b4b4269889028549 2023-04-05 18:56:12,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/322a469e1a4f4a15b4b4269889028549, entries=150, sequenceid=467, filesize=11.9 K 2023-04-05 18:56:12,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/3e8f4b7284c94a93b6deccafb4ec9cc7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3e8f4b7284c94a93b6deccafb4ec9cc7 2023-04-05 18:56:12,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3e8f4b7284c94a93b6deccafb4ec9cc7, entries=150, sequenceid=467, filesize=11.9 K 2023-04-05 18:56:12,315 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 477ms, sequenceid=467, compaction requested=true 2023-04-05 18:56:12,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:12,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:12,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:12,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:56:12,316 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-05 18:56:12,316 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-05 18:56:12,316 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-05 18:56:12,316 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. because compaction request was cancelled 2023-04-05 18:56:12,317 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:12,317 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:12,317 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:12,317 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,317 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/3597d99588d04002bf0cd432f90e6e55, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1c531d22eca449b0b07d5722f5fbc889, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/322a469e1a4f4a15b4b4269889028549] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.1 K 2023-04-05 18:56:12,318 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 3597d99588d04002bf0cd432f90e6e55, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680720970698 2023-04-05 18:56:12,318 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1c531d22eca449b0b07d5722f5fbc889, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680720970706 2023-04-05 18:56:12,319 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 322a469e1a4f4a15b4b4269889028549, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720971302 2023-04-05 18:56:12,337 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#563 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:12,349 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/9d701842e2724f5f96105b9bbbcf0d5b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/9d701842e2724f5f96105b9bbbcf0d5b 2023-04-05 18:56:12,352 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 9d701842e2724f5f96105b9bbbcf0d5b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:12,352 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:12,352 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=13, startTime=1680720972316; duration=0sec 2023-04-05 18:56:12,352 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:12,352 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:12,355 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:12,355 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction (all files) 2023-04-05 18:56:12,355 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,356 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/1eb2e363ca7e4777a973f4543fba68b5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c0dd86d9676e4bb99ba03287fc65552c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3e8f4b7284c94a93b6deccafb4ec9cc7] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.1 K 2023-04-05 18:56:12,356 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1eb2e363ca7e4777a973f4543fba68b5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680720970698 2023-04-05 18:56:12,356 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting c0dd86d9676e4bb99ba03287fc65552c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680720970706 2023-04-05 18:56:12,356 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 3e8f4b7284c94a93b6deccafb4ec9cc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720971302 2023-04-05 18:56:12,371 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#564 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:12,375 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:12,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:12,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:12,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:12,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:12,385 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:12,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:12,385 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:12,386 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,386 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,387 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,387 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,387 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,387 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:12,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:12,388 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:12,388 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,388 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:12,388 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:12,388 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:12,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:12,389 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 18:56:12,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:12,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:12,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:12,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:12,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:12,389 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:12,407 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304054e987f37d0fe45ff9c1ede7f233d0439 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054e987f37d0fe45ff9c1ede7f233d0439 2023-04-05 18:56:12,413 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=482, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/05362fe9a750426f866d5a0ba35fc4ba 2023-04-05 18:56:12,424 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/cab4f353ca0a44da810ad1ddb72b1adb 2023-04-05 18:56:12,437 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/6fff6379679b4df3bba50fdcbd19491b 2023-04-05 18:56:12,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:12,440 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:12,440 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/05362fe9a750426f866d5a0ba35fc4ba as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/05362fe9a750426f866d5a0ba35fc4ba 2023-04-05 18:56:12,444 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/05362fe9a750426f866d5a0ba35fc4ba, entries=150, sequenceid=482, filesize=25.4 K 2023-04-05 18:56:12,445 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/cab4f353ca0a44da810ad1ddb72b1adb as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/cab4f353ca0a44da810ad1ddb72b1adb 2023-04-05 18:56:12,448 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/cab4f353ca0a44da810ad1ddb72b1adb, entries=150, sequenceid=482, filesize=11.9 K 2023-04-05 18:56:12,449 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/6fff6379679b4df3bba50fdcbd19491b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6fff6379679b4df3bba50fdcbd19491b 2023-04-05 18:56:12,453 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6fff6379679b4df3bba50fdcbd19491b, entries=150, sequenceid=482, filesize=11.9 K 2023-04-05 18:56:12,454 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/fd1257cddc1a4a178626e19fcd6d9cc9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fd1257cddc1a4a178626e19fcd6d9cc9 2023-04-05 18:56:12,455 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=20.13 KB/20610 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 66ms, sequenceid=482, compaction requested=true 2023-04-05 18:56:12,455 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:12,455 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:12,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:12,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:12,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:12,458 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,458 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,459 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,460 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/A of 7a7f1bea453fa17f7cd1e6eff9f4a686 into fd1257cddc1a4a178626e19fcd6d9cc9(size=25.8 K), total size for store is 109.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 18:56:12,460 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:12,460 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/A, priority=12, startTime=1680720971179; duration=1sec 2023-04-05 18:56:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,460 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:12,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:12,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11fa7f67[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:12,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:12,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,466 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,466 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:12,466 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:12,466 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:12,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:12,468 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,468 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:12,468 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,468 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:12,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:12,475 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:12,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:12,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:12,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:12,475 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:12,476 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,476 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,476 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,477 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:12,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,480 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:12,480 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:12,480 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:12,480 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:12,480 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:12,480 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,482 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,488 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,488 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,488 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,488 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:12,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:12,488 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:12,488 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:12,488 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,489 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,488 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,489 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,489 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,489 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,498 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040582cb096be40040639f66ab006ec7b356 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040582cb096be40040639f66ab006ec7b356 2023-04-05 18:56:12,502 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=494, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/c5b9e1c60db64f9f944df80939f1bdf4 2023-04-05 18:56:12,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/c8524ecb747c4ea9893cffd3b1a9470f 2023-04-05 18:56:12,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/37e118a448454d6cbffdc175d2d2798a 2023-04-05 18:56:12,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680720972512, max timestamp=1680720972524Num uniques 859; 2023-04-05 18:56:12,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 18:56:12,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720972512, max timestamp=1680720972524Num uniques 859; 2023-04-05 18:56:12,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 18:56:12,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680720972512, max timestamp=1680720972524Num uniques 859; 2023-04-05 18:56:12,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 18:56:12,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/c5b9e1c60db64f9f944df80939f1bdf4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c5b9e1c60db64f9f944df80939f1bdf4 2023-04-05 18:56:12,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c5b9e1c60db64f9f944df80939f1bdf4, entries=200, sequenceid=494, filesize=32.3 K 2023-04-05 18:56:12,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/c8524ecb747c4ea9893cffd3b1a9470f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/c8524ecb747c4ea9893cffd3b1a9470f 2023-04-05 18:56:12,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/c8524ecb747c4ea9893cffd3b1a9470f, entries=100, sequenceid=494, filesize=9.5 K 2023-04-05 18:56:12,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/37e118a448454d6cbffdc175d2d2798a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/37e118a448454d6cbffdc175d2d2798a 2023-04-05 18:56:12,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/37e118a448454d6cbffdc175d2d2798a, entries=100, sequenceid=494, filesize=9.5 K 2023-04-05 18:56:12,544 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=86.43 KB/88500 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 65ms, sequenceid=494, compaction requested=true 2023-04-05 18:56:12,544 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:12,545 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:12,545 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 18:56:12,545 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:12,545 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:56:12,546 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 144682 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 18:56:12,546 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/A is initiating minor compaction (all files) 2023-04-05 18:56:12,546 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/A in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,546 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fd1257cddc1a4a178626e19fcd6d9cc9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/e4ec620d4e6145558ae8512eace0b0de, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/09efbb489451490eb0b08a53197bf224, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/05362fe9a750426f866d5a0ba35fc4ba, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c5b9e1c60db64f9f944df80939f1bdf4] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=141.3 K 2023-04-05 18:56:12,547 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting fd1257cddc1a4a178626e19fcd6d9cc9, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680720970698 2023-04-05 18:56:12,547 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting e4ec620d4e6145558ae8512eace0b0de, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680720970706 2023-04-05 18:56:12,547 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 09efbb489451490eb0b08a53197bf224, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720971302 2023-04-05 18:56:12,547 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 05362fe9a750426f866d5a0ba35fc4ba, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1680720971849 2023-04-05 18:56:12,548 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c5b9e1c60db64f9f944df80939f1bdf4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680720972458 2023-04-05 18:56:12,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:12,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=126.68 KB heapSize=363.23 KB 2023-04-05 18:56:12,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:12,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:12,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:12,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:12,557 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:12,557 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:12,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721032563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721032568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721032569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,577 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#A#compaction#571 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:12,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721032572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,589 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:12,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:12,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:12,595 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405dbd2d4700af747c2a074c8a063163113 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405dbd2d4700af747c2a074c8a063163113 2023-04-05 18:56:12,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:12,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:12,600 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:12,600 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,600 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:12,600 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:12,601 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,601 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,603 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,604 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,604 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:12,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:12,612 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:12,612 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,612 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:12,612 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:12,612 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:12,612 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,613 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:12,613 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:12,613 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:12,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:12,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:12,613 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:12,614 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=530, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/71739b1eb82a44288aae3b8932324e29 2023-04-05 18:56:12,614 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,614 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,615 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:12,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a997b16[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:12,618 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:12,618 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,620 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,620 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:12,620 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:12,620 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:12,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,620 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:12,621 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:12,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,621 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:12,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:12,625 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:12,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:12,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:12,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:12,625 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:12,628 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/61549a07255c4399b1fdb7c55d5093f0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/61549a07255c4399b1fdb7c55d5093f0 2023-04-05 18:56:12,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/933b438eaeb54d22be1f3a47f0b09e1b 2023-04-05 18:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,629 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,631 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,632 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/A of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 61549a07255c4399b1fdb7c55d5093f0(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,632 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:12,633 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/A, priority=11, startTime=1680720972545; duration=0sec 2023-04-05 18:56:12,633 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:56:12,633 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-05 18:56:12,633 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,633 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,633 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,633 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,634 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-05 18:56:12,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:12,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:12,634 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-05 18:56:12,634 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. because compaction request was cancelled 2023-04-05 18:56:12,634 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,634 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:12,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,634 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,634 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:12,634 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:12,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:12,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,636 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34592 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:12,636 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:12,636 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,636 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/9d701842e2724f5f96105b9bbbcf0d5b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/cab4f353ca0a44da810ad1ddb72b1adb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/c8524ecb747c4ea9893cffd3b1a9470f] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=33.8 K 2023-04-05 18:56:12,637 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 9d701842e2724f5f96105b9bbbcf0d5b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720971302 2023-04-05 18:56:12,638 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting cab4f353ca0a44da810ad1ddb72b1adb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1680720971849 2023-04-05 18:56:12,638 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c8524ecb747c4ea9893cffd3b1a9470f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680720972478 2023-04-05 18:56:12,640 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/fc3e3d8abd56406492f1b3342a5e33f0 2023-04-05 18:56:12,644 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#575 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:12,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/71739b1eb82a44288aae3b8932324e29 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71739b1eb82a44288aae3b8932324e29 2023-04-05 18:56:12,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71739b1eb82a44288aae3b8932324e29, entries=150, sequenceid=530, filesize=25.4 K 2023-04-05 18:56:12,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/933b438eaeb54d22be1f3a47f0b09e1b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/933b438eaeb54d22be1f3a47f0b09e1b 2023-04-05 18:56:12,658 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/54d311c7981843bf879bb296eb69b20e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/54d311c7981843bf879bb296eb69b20e 2023-04-05 18:56:12,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/933b438eaeb54d22be1f3a47f0b09e1b, entries=150, sequenceid=530, filesize=11.9 K 2023-04-05 18:56:12,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/fc3e3d8abd56406492f1b3342a5e33f0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fc3e3d8abd56406492f1b3342a5e33f0 2023-04-05 18:56:12,664 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 54d311c7981843bf879bb296eb69b20e(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:12,664 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:12,664 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=13, startTime=1680720972545; duration=0sec 2023-04-05 18:56:12,665 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:12,666 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fc3e3d8abd56406492f1b3342a5e33f0, entries=150, sequenceid=530, filesize=11.9 K 2023-04-05 18:56:12,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~126.68 KB/129720, heapSize ~363.19 KB/371904, currentSize=60.38 KB/61830 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 112ms, sequenceid=530, compaction requested=true 2023-04-05 18:56:12,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:12,667 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:12,667 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-05 18:56:12,667 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:12,667 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 18:56:12,668 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-05 18:56:12,668 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-05 18:56:12,668 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. because compaction request was cancelled 2023-04-05 18:56:12,668 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 3 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:12,670 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34114 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:12,670 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction 2023-04-05 18:56:12,670 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,670 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6fff6379679b4df3bba50fdcbd19491b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/37e118a448454d6cbffdc175d2d2798a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fc3e3d8abd56406492f1b3342a5e33f0] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=33.3 K 2023-04-05 18:56:12,671 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 6fff6379679b4df3bba50fdcbd19491b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482 2023-04-05 18:56:12,671 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 37e118a448454d6cbffdc175d2d2798a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=494 2023-04-05 18:56:12,671 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting fc3e3d8abd56406492f1b3342a5e33f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530 2023-04-05 18:56:12,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:12,673 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 18:56:12,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:12,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:12,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:12,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:12,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:12,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:12,684 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#577 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:12,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721032691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721032692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721032692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721032693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721032694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,722 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/27a7fe3cec404d19af56e912626449d1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/27a7fe3cec404d19af56e912626449d1 2023-04-05 18:56:12,725 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 27a7fe3cec404d19af56e912626449d1(size=12.0 K), total size for store is 48.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:12,725 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:12,726 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=10, startTime=1680720972667; duration=0sec 2023-04-05 18:56:12,726 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:12,726 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-05 18:56:12,726 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-05 18:56:12,726 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-05 18:56:12,726 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. because compaction request was cancelled 2023-04-05 18:56:12,734 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:12,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:12,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:12,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:12,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:12,738 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:12,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:12,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:12,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,738 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,739 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,740 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,740 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:12,741 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:12,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:12,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:12,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:12,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:12,743 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:12,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:12,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:12,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,743 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:12,743 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:12,743 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,743 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:12,743 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:12,745 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,745 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,745 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,745 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749ad39b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:12,747 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:12,747 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:12,751 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,751 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,753 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:12,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:12,756 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:12,756 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:12,756 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:12,756 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:12,756 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:12,757 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,757 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,758 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,758 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,760 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,761 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,762 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:12,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:12,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,762 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:12,762 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:12,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,762 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,791 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/fe161307cf6a4e6e9a845758183afc53 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fe161307cf6a4e6e9a845758183afc53 2023-04-05 18:56:12,797 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into fe161307cf6a4e6e9a845758183afc53(size=12.4 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:12,797 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:12,797 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=13, startTime=1680720972316; duration=0sec 2023-04-05 18:56:12,798 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:12,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721032796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721032796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721032796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721032798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721032798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,862 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:12,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:12,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:12,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:12,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:12,865 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:12,865 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,865 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:12,865 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:12,866 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,866 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,867 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,867 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,867 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,868 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:12,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:12,869 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,869 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:12,869 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:12,869 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:12,872 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bf33067[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:12,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:12,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,878 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:12,878 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,878 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:12,879 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:12,881 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:12,881 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:12,881 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:12,881 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:12,881 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:12,883 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,883 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:12,883 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,883 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:12,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,887 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,887 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,888 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:12,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:12,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:12,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:12,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:12,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:12,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,888 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:12,888 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:12,888 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:12,989 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:12,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:12,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:12,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:12,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:12,993 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:12,993 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,993 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:12,993 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:12,994 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,994 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,995 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:12,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:12,995 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,995 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:12,996 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:12,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:12,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:13,000 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:13,000 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:13,000 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:13,000 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:13,000 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:13,001 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:13,001 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:13,001 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:13,001 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,001 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:13,001 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:13,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721032999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721033000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721033000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721033001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,002 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721033002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:13,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:13,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b07f39[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:13,004 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:13,004 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,005 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:13,006 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,006 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:13,008 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:13,008 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:13,008 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:13,008 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:13,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:13,012 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:13,012 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:13,012 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:13,012 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:13,012 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:13,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,013 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:13,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:13,014 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,015 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,017 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:13,017 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:13,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:13,017 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,017 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:13,017 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:13,017 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:13,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:13,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:13,017 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,018 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,018 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:13,018 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:13,018 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:13,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:13,018 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,109 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040539270aa38a9f4eabae7e7bf7e552334d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040539270aa38a9f4eabae7e7bf7e552334d 2023-04-05 18:56:13,118 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:13,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:13,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:13,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:13,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:13,122 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:13,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:13,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:13,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:13,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:13,123 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:13,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:13,124 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=545, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/c64a78de0e0841ed9cc398ea9dc7f7af 2023-04-05 18:56:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:13,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,124 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:13,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:13,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:13,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:13,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:13,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:13,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:13,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:13,126 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:13,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:13,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:13,126 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:13,127 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:13,127 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:13,127 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:13,127 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,127 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:13,127 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:13,129 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:13,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,129 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:13,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/e0bbb341488442b996bcd48c157306af 2023-04-05 18:56:13,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:13,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5820f4d2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:13,141 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:13,141 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,143 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,143 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:13,143 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:13,143 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:13,143 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,143 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:13,148 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:13,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:13,151 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:13,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:13,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:13,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:13,151 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:13,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/63ef95ea1f614270a0a0c3fd521f8347 2023-04-05 18:56:13,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:13,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:13,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,154 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/c64a78de0e0841ed9cc398ea9dc7f7af as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c64a78de0e0841ed9cc398ea9dc7f7af 2023-04-05 18:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:13,157 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,159 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:13,159 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:13,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:13,159 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:13,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:13,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:13,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:13,160 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:13,160 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,160 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:13,160 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:13,160 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:13,160 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,160 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,160 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:13,160 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,162 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c64a78de0e0841ed9cc398ea9dc7f7af, entries=200, sequenceid=545, filesize=32.3 K 2023-04-05 18:56:13,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/e0bbb341488442b996bcd48c157306af as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/e0bbb341488442b996bcd48c157306af 2023-04-05 18:56:13,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/e0bbb341488442b996bcd48c157306af, entries=150, sequenceid=545, filesize=11.9 K 2023-04-05 18:56:13,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/63ef95ea1f614270a0a0c3fd521f8347 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/63ef95ea1f614270a0a0c3fd521f8347 2023-04-05 18:56:13,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/63ef95ea1f614270a0a0c3fd521f8347, entries=150, sequenceid=545, filesize=11.9 K 2023-04-05 18:56:13,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 504ms, sequenceid=545, compaction requested=true 2023-04-05 18:56:13,177 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:13,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:13,177 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:13,177 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:13,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:13,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:13,178 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85724 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:13,178 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/A is initiating minor compaction (all files) 2023-04-05 18:56:13,178 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:13,178 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/A in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:13,178 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:13,178 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/61549a07255c4399b1fdb7c55d5093f0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71739b1eb82a44288aae3b8932324e29, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c64a78de0e0841ed9cc398ea9dc7f7af] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=83.7 K 2023-04-05 18:56:13,178 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:13,178 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/54d311c7981843bf879bb296eb69b20e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/933b438eaeb54d22be1f3a47f0b09e1b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/e0bbb341488442b996bcd48c157306af] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.3 K 2023-04-05 18:56:13,178 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 61549a07255c4399b1fdb7c55d5093f0, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680720971861 2023-04-05 18:56:13,179 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 54d311c7981843bf879bb296eb69b20e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680720971861 2023-04-05 18:56:13,179 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 71739b1eb82a44288aae3b8932324e29, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680720972535 2023-04-05 18:56:13,179 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 933b438eaeb54d22be1f3a47f0b09e1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680720972535 2023-04-05 18:56:13,179 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting c64a78de0e0841ed9cc398ea9dc7f7af, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680720972559 2023-04-05 18:56:13,179 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting e0bbb341488442b996bcd48c157306af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680720972567 2023-04-05 18:56:13,188 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#581 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:13,191 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#A#compaction#580 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:13,201 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/95fc341adb5840bfb46fdfbb41624ced as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/95fc341adb5840bfb46fdfbb41624ced 2023-04-05 18:56:13,207 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 95fc341adb5840bfb46fdfbb41624ced(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:13,207 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:13,207 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=13, startTime=1680720973177; duration=0sec 2023-04-05 18:56:13,207 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:13,207 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:13,209 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37139 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:13,209 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction (all files) 2023-04-05 18:56:13,209 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:13,209 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fe161307cf6a4e6e9a845758183afc53, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/27a7fe3cec404d19af56e912626449d1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/63ef95ea1f614270a0a0c3fd521f8347] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.3 K 2023-04-05 18:56:13,209 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting fe161307cf6a4e6e9a845758183afc53, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680720971302 2023-04-05 18:56:13,210 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 27a7fe3cec404d19af56e912626449d1, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680720972535 2023-04-05 18:56:13,211 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 63ef95ea1f614270a0a0c3fd521f8347, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680720972567 2023-04-05 18:56:13,216 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#582 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:13,234 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/21ddc171957a4204a180e2cc45bd5eb5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21ddc171957a4204a180e2cc45bd5eb5 2023-04-05 18:56:13,240 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 21ddc171957a4204a180e2cc45bd5eb5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:13,240 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:13,240 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=13, startTime=1680720973177; duration=0sec 2023-04-05 18:56:13,240 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:13,260 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:13,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:13,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:13,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:13,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:13,265 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:13,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:13,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:13,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:13,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,265 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:13,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,267 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:13,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:13,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:13,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:13,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:13,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:13,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:13,270 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:13,270 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:13,270 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 18:56:13,270 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:13,270 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:13,270 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:13,270 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:13,271 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:13,271 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:13,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:13,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:13,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721033310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721033312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721033313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721033313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721033314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721033415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721033415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721033418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721033418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721033418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721033618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721033619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721033620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721033621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721033621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,698 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405b8ec0a046e594ac29607e54c44b175f0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b8ec0a046e594ac29607e54c44b175f0 2023-04-05 18:56:13,729 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=568, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/fdc76454546643748ad801eab6bfa9b7 2023-04-05 18:56:13,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721033922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56746 deadline: 1680721033923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721033924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721033925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:13,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:13,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721033926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,044 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/60da56b565b34026a9ca7b94b8fbe88a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60da56b565b34026a9ca7b94b8fbe88a 2023-04-05 18:56:14,047 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/A of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 60da56b565b34026a9ca7b94b8fbe88a(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:14,047 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:14,047 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/A, priority=13, startTime=1680720973177; duration=0sec 2023-04-05 18:56:14,047 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:14,140 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/66ca9482712946179b5ca8bdcc4701f3 2023-04-05 18:56:14,170 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/c5b73a7161ba4844887b129966f96a6d 2023-04-05 18:56:14,174 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/fdc76454546643748ad801eab6bfa9b7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fdc76454546643748ad801eab6bfa9b7 2023-04-05 18:56:14,177 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fdc76454546643748ad801eab6bfa9b7, entries=150, sequenceid=568, filesize=25.4 K 2023-04-05 18:56:14,177 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/66ca9482712946179b5ca8bdcc4701f3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/66ca9482712946179b5ca8bdcc4701f3 2023-04-05 18:56:14,181 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/66ca9482712946179b5ca8bdcc4701f3, entries=150, sequenceid=568, filesize=11.9 K 2023-04-05 18:56:14,183 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/c5b73a7161ba4844887b129966f96a6d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c5b73a7161ba4844887b129966f96a6d 2023-04-05 18:56:14,187 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c5b73a7161ba4844887b129966f96a6d, entries=150, sequenceid=568, filesize=11.9 K 2023-04-05 18:56:14,188 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 918ms, sequenceid=568, compaction requested=false 2023-04-05 18:56:14,188 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:14,188 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:14,190 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,190 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:14,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:14,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:14,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:14,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a84e246[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:14,193 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:14,194 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:14,195 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:14,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:14,195 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:14,195 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:14,195 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:14,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:14,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:14,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:14,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:14,199 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:14,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:14,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:14,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:14,199 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:14,201 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:14,201 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:14,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:14,201 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:14,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:14,202 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:14,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:14,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:14,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:14,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:14,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:14,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:14,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:14,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:14,206 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:14,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:14,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:14,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:14,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:14,206 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:14,206 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:14,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:14,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:14,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:14,307 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:14,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:14,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:14,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:14,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:14,310 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:14,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:14,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:14,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:14,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:14,310 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:14,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,312 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:14,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:14,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:14,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:14,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:14,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:14,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:14,314 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 18:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:14,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:14,428 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:14,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721034442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721034442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721034445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721034446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721034446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721034547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721034547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721034549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721034549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721034549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721034748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721034749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721034752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721034753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:14,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721034753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:14,777 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304057901226bd5374230a4d1ce6afca7abae to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304057901226bd5374230a4d1ce6afca7abae 2023-04-05 18:56:14,800 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=582, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/d59ed6963be346f09e604fc19ca40a96 2023-04-05 18:56:15,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721035052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721035053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721035054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721035054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721035056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,213 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/6503a6e49c074fed8482cdd67df46afd 2023-04-05 18:56:15,242 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/67393ef8c2e944b992af5c2136dbcbaa 2023-04-05 18:56:15,246 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/d59ed6963be346f09e604fc19ca40a96 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/d59ed6963be346f09e604fc19ca40a96 2023-04-05 18:56:15,251 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/d59ed6963be346f09e604fc19ca40a96, entries=150, sequenceid=582, filesize=25.4 K 2023-04-05 18:56:15,253 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/6503a6e49c074fed8482cdd67df46afd as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6503a6e49c074fed8482cdd67df46afd 2023-04-05 18:56:15,257 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6503a6e49c074fed8482cdd67df46afd, entries=150, sequenceid=582, filesize=11.9 K 2023-04-05 18:56:15,258 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/67393ef8c2e944b992af5c2136dbcbaa as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/67393ef8c2e944b992af5c2136dbcbaa 2023-04-05 18:56:15,263 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/67393ef8c2e944b992af5c2136dbcbaa, entries=150, sequenceid=582, filesize=11.9 K 2023-04-05 18:56:15,264 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 950ms, sequenceid=582, compaction requested=true 2023-04-05 18:56:15,264 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:15,264 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:15,269 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,269 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,270 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,270 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:15,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:15,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:15,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:15,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:15,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28269579[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:15,272 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:15,272 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:15,275 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,275 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:15,278 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:15,279 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:15,279 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:15,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:15,288 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:15,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:15,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:15,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:15,288 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:15,291 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:15,291 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,291 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:15,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,295 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,298 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,306 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:15,306 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:15,306 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:15,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:15,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:15,307 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:15,307 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:15,306 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:15,306 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:15,307 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:15,307 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,307 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,307 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,307 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:15,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:15,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:15,407 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:15,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:15,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:15,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:15,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:15,416 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:15,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:15,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:15,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:15,417 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,418 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:15,424 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:15,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:15,424 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:15,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:15,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:15,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:15,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:15,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:15,426 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:15,426 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:15,427 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:56:15,427 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:15,427 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:15,427 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:15,427 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:15,427 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:15,427 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:15,475 WARN [DataStreamer for file /user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405b6e6f8dfc18d422298a4c80d02bf86d1] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-05 18:56:15,489 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405b6e6f8dfc18d422298a4c80d02bf86d1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b6e6f8dfc18d422298a4c80d02bf86d1 2023-04-05 18:56:15,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:15,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:15,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721035563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721035563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721035564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721035565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721035565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721035666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721035667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721035667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721035667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721035668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721035869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721035870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721035870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721035871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:15,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721035872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,934 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=602, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/7bde8e4f3b4c4212852cc60dfdc74a10 2023-04-05 18:56:15,944 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/4114e60a99a9440180c91ce1e4348f1d 2023-04-05 18:56:15,953 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/924e5225e6654eae91f2a051a2f3f481 2023-04-05 18:56:15,958 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/7bde8e4f3b4c4212852cc60dfdc74a10 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7bde8e4f3b4c4212852cc60dfdc74a10 2023-04-05 18:56:15,961 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7bde8e4f3b4c4212852cc60dfdc74a10, entries=150, sequenceid=602, filesize=25.4 K 2023-04-05 18:56:15,962 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/4114e60a99a9440180c91ce1e4348f1d as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4114e60a99a9440180c91ce1e4348f1d 2023-04-05 18:56:15,969 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4114e60a99a9440180c91ce1e4348f1d, entries=150, sequenceid=602, filesize=11.9 K 2023-04-05 18:56:15,970 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/924e5225e6654eae91f2a051a2f3f481 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/924e5225e6654eae91f2a051a2f3f481 2023-04-05 18:56:15,974 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/924e5225e6654eae91f2a051a2f3f481, entries=150, sequenceid=602, filesize=11.9 K 2023-04-05 18:56:15,974 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 547ms, sequenceid=602, compaction requested=true 2023-04-05 18:56:15,974 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:15,974 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:15,975 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:15,975 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:15,975 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,975 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:15,975 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:15,976 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,976 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:15,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:15,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22e57ef3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:15,979 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:15,979 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,980 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,980 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:15,980 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:15,980 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:15,981 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,981 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:15,984 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,984 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:15,984 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:15,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:15,987 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:15,987 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:15,987 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:15,987 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:15,987 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:15,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:15,989 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:15,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,991 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:15,992 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:15,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:15,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:15,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:15,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:15,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:15,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:15,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:15,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:15,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:15,998 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:15,998 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:15,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:15,998 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:16,099 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:16,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:16,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:16,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:16,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:16,134 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:16,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:16,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:16,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:16,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:16,134 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:16,138 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:16,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:16,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:16,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,139 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:16,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:16,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:16,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:16,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:16,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:16,148 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:16,148 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:16,148 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:16,160 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:16,160 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:16,164 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:16,164 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:16,164 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:56:16,164 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:16,164 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:16,165 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:16,165 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:16,165 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:16,165 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:16,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:16,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:16,186 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d5581b41f47a4bffa68ecd1cd8d9d35f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d5581b41f47a4bffa68ecd1cd8d9d35f 2023-04-05 18:56:16,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721036191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721036190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721036193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721036194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721036196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721036297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721036297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721036297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721036297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721036300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721036500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721036500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721036501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721036501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721036503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,597 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=614, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/a91856386dd140a8b53747ca37da7e5a 2023-04-05 18:56:16,606 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/6f1db48b7fd54c828ba8ea4714eb8387 2023-04-05 18:56:16,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721036802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721036804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721036804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721036805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:16,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:16,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721036806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,014 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/3361f54289dd4646963b865fe7b8478f 2023-04-05 18:56:17,020 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/a91856386dd140a8b53747ca37da7e5a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a91856386dd140a8b53747ca37da7e5a 2023-04-05 18:56:17,023 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a91856386dd140a8b53747ca37da7e5a, entries=150, sequenceid=614, filesize=25.4 K 2023-04-05 18:56:17,023 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/6f1db48b7fd54c828ba8ea4714eb8387 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6f1db48b7fd54c828ba8ea4714eb8387 2023-04-05 18:56:17,026 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6f1db48b7fd54c828ba8ea4714eb8387, entries=150, sequenceid=614, filesize=11.9 K 2023-04-05 18:56:17,027 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/3361f54289dd4646963b865fe7b8478f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3361f54289dd4646963b865fe7b8478f 2023-04-05 18:56:17,030 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3361f54289dd4646963b865fe7b8478f, entries=150, sequenceid=614, filesize=11.9 K 2023-04-05 18:56:17,030 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 866ms, sequenceid=614, compaction requested=true 2023-04-05 18:56:17,030 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:17,030 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:17,030 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:17,030 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:17,032 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:17,032 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:17,033 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:17,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:17,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:17,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:17,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dbc9819[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:17,034 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:17,034 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,035 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,035 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:17,035 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,035 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:17,035 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:17,035 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:17,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:17,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:17,036 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,036 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:17,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:17,038 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:17,039 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:17,039 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:17,039 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:17,039 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:17,041 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,041 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:17,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,041 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:17,042 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:17,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:17,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,045 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:17,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:17,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:17,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:17,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:17,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:17,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:17,045 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:17,045 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:17,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:17,145 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:17,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:17,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:17,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:17,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:17,151 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:17,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:17,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:17,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:17,152 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,152 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,153 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:17,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:17,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:17,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:17,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:17,156 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:17,156 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:17,156 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:17,168 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:17,168 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:17,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:17,168 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:17,184 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:56:17,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:17,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:17,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:17,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:17,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:17,184 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:17,212 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405c0d828ad70ab4fd58b8872abf483ce0a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c0d828ad70ab4fd58b8872abf483ce0a 2023-04-05 18:56:17,224 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=634, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/f8e761340e1f4a4390c76cae88a7cd83 2023-04-05 18:56:17,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:17,311 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:17,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721037317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721037319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721037320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721037320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721037324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721037423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721037423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721037425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721037425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721037426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721037626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721037627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721037631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721037631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:17,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721037632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,638 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/a34491dd6639490d814d8e415e29628f 2023-04-05 18:56:17,662 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/b70554e2c92d4ff0adc6e1d196c3b155 2023-04-05 18:56:17,669 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/f8e761340e1f4a4390c76cae88a7cd83 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f8e761340e1f4a4390c76cae88a7cd83 2023-04-05 18:56:17,674 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f8e761340e1f4a4390c76cae88a7cd83, entries=150, sequenceid=634, filesize=25.4 K 2023-04-05 18:56:17,675 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/a34491dd6639490d814d8e415e29628f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a34491dd6639490d814d8e415e29628f 2023-04-05 18:56:17,679 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a34491dd6639490d814d8e415e29628f, entries=150, sequenceid=634, filesize=11.9 K 2023-04-05 18:56:17,680 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/b70554e2c92d4ff0adc6e1d196c3b155 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b70554e2c92d4ff0adc6e1d196c3b155 2023-04-05 18:56:17,684 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b70554e2c92d4ff0adc6e1d196c3b155, entries=150, sequenceid=634, filesize=11.9 K 2023-04-05 18:56:17,684 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 500ms, sequenceid=634, compaction requested=true 2023-04-05 18:56:17,684 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:17,684 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:17,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:17,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:17,685 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:17,687 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,687 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:17,688 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,688 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2077c33a[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:17,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:17,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,691 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:17,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:17,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:17,693 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:17,693 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:17,693 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:17,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:17,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:17,694 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:17,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:17,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:17,694 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:17,694 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:17,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:17,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:17,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,696 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:17,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:17,697 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,701 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:17,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:17,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:17,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:17,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:17,702 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:17,702 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:17,701 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:17,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:17,802 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:17,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:17,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:17,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:17,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:17,806 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:17,806 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:17,806 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:17,806 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:17,806 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,806 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,808 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:17,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:17,809 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,809 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:17,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:17,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:17,810 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:17,810 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:17,811 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 18:56:17,811 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:17,811 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:17,811 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:17,811 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:17,811 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:17,811 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:17,824 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304055d9a89af5d554965b86fd5b196f1a872 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055d9a89af5d554965b86fd5b196f1a872 2023-04-05 18:56:17,829 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=647, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/bf04fead67c7404093c1bfe5cf4b9962 2023-04-05 18:56:17,837 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=647 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/735e27a5a4064896bdefb428ab8b351b 2023-04-05 18:56:17,848 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=647 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/38c5012e9e534ae493cfd487ea2833f2 2023-04-05 18:56:17,853 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/bf04fead67c7404093c1bfe5cf4b9962 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/bf04fead67c7404093c1bfe5cf4b9962 2023-04-05 18:56:17,860 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/bf04fead67c7404093c1bfe5cf4b9962, entries=150, sequenceid=647, filesize=25.4 K 2023-04-05 18:56:17,861 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/735e27a5a4064896bdefb428ab8b351b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/735e27a5a4064896bdefb428ab8b351b 2023-04-05 18:56:17,866 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/735e27a5a4064896bdefb428ab8b351b, entries=150, sequenceid=647, filesize=11.9 K 2023-04-05 18:56:17,866 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/38c5012e9e534ae493cfd487ea2833f2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/38c5012e9e534ae493cfd487ea2833f2 2023-04-05 18:56:17,870 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/38c5012e9e534ae493cfd487ea2833f2, entries=150, sequenceid=647, filesize=11.9 K 2023-04-05 18:56:17,870 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 59ms, sequenceid=647, compaction requested=true 2023-04-05 18:56:17,870 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:17,870 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:17,872 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,872 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:17,873 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,873 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:17,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:17,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:17,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:17,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:17,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c660710[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:17,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:17,875 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,876 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:17,876 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:17,878 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:17,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:17,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:17,878 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:17,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:17,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:17,885 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:17,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:17,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:17,885 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:17,885 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:17,887 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,887 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:17,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,887 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:17,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:17,888 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:17,889 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:17,893 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:17,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:17,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:17,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,894 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:17,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:17,894 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:17,894 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:17,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:17,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:17,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:17,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:17,894 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:17,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:17,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:17,958 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:56:17,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:17,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:17,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:17,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:17,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:17,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:17,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680720977968, max timestamp=1680720977985Num uniques 859; 2023-04-05 18:56:17,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 18:56:17,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680720977968, max timestamp=1680720977985Num uniques 859; 2023-04-05 18:56:17,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 18:56:17,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680720977968, max timestamp=1680720977985Num uniques 859; 2023-04-05 18:56:17,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-05 18:56:17,994 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:17,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:17,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:17,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:17,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:17,997 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:17,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:17,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:17,997 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:17,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:17,998 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,000 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,000 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,001 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,001 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:18,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:18,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:18,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:18,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:18,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:18,002 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,002 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:18,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:18,002 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:18,002 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:18,002 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:18,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:18,003 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:18,004 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,004 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@357df489[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:18,006 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:18,006 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,007 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,007 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:18,007 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:18,007 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:18,007 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,007 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:18,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721038001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721038002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,008 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721038003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721038005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721038008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:18,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:18,014 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:18,014 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:18,014 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:18,014 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:18,014 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:18,016 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,016 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,016 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,021 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,022 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,024 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,024 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,024 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,024 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,024 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,024 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,024 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,024 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:18,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:18,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:18,024 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,025 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,025 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:18,025 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:18,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721038109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721038109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721038111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721038112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721038114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,125 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:18,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:18,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:18,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:18,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:18,130 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:18,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:18,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:18,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,131 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,132 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,132 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,133 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,133 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:18,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:18,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:18,134 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:18,135 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,135 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:18,135 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:18,135 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:18,138 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,138 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,138 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,138 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:18,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c18de6f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:18,140 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:18,140 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,141 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,142 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:18,142 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:18,142 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:18,142 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,142 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:18,143 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,143 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:18,143 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,143 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:18,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:18,146 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:18,146 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:18,146 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:18,146 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:18,146 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:18,149 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,149 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,149 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,151 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,153 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,155 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:18,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:18,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,155 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,155 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:18,155 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:18,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,255 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:18,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:18,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:18,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:18,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:18,267 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:18,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:18,267 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:18,268 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,268 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,269 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,269 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,270 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,270 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:18,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:18,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:18,272 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:18,272 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,272 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:18,273 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,273 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:18,273 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:18,273 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:18,275 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,275 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,276 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,276 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3268a6c5[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:18,277 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:18,277 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,278 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:18,278 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,278 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:18,280 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:18,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,280 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:18,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:18,282 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:18,282 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:18,282 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:18,282 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:18,282 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:18,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,285 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,286 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,288 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,288 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,288 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,288 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,288 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,288 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:18,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,288 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:18,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:18,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,288 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,288 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:18,289 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:18,289 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,289 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721038313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721038313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721038316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721038318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721038318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,385 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040554ea51466ee8440cae5fc91c44aa0670 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040554ea51466ee8440cae5fc91c44aa0670 2023-04-05 18:56:18,389 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:18,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:18,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:18,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:18,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:18,412 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:18,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:18,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:18,412 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,413 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,417 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,417 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:18,418 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,418 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:18,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:18,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:18,419 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,419 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:18,419 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:18,419 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:18,422 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,422 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,422 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,422 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e3a791f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:18,423 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:18,423 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,424 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:18,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:18,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,425 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:18,425 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:18,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:18,426 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,426 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:18,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:18,428 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:18,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:18,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:18,428 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:18,428 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:18,430 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,430 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,430 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,431 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,432 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,436 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,436 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,436 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:18,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,437 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:18,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:18,437 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,438 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:18,438 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:18,538 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:18,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:18,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:18,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:18,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:18,541 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:18,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:18,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:18,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,542 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,542 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,543 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,543 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:18,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:18,544 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:18,544 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:18,544 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:18,544 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:18,545 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,545 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64391bed[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:18,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:18,547 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,548 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:18,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:18,551 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,551 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,551 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,551 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:18,553 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:18,553 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:18,553 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:18,553 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:18,553 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:18,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,555 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,555 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,556 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,559 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:18,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:18,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,559 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,560 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:18,560 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:18,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721038618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721038620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721038621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721038622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:18,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721038624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,660 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:18,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:18,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:18,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:18,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:18,663 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:18,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:18,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:18,664 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,664 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,665 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,665 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,665 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,665 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:18,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:18,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:18,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,673 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:18,681 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,681 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:18,682 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:18,682 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:18,683 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,683 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e384aa3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:18,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:18,690 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,691 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:18,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:18,692 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,692 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:18,694 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:18,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:18,698 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:18,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:18,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:18,698 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:18,698 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:18,699 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,699 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,699 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,700 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,702 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,704 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:18,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:18,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:18,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,704 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,705 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:18,705 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:18,805 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:18,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:18,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:18,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:18,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:18,808 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:18,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:18,808 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:18,809 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,809 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:18,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:18,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:18,811 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,812 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:18,812 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:18,812 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:18,813 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,813 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,814 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,814 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=658, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/0fb3cfdaf9ca43efbe8752d8e147377c 2023-04-05 18:56:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:18,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79119b2f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:18,815 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:18,815 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,820 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,821 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:18,821 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:18,821 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:18,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,821 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:18,822 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,825 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/4c1af5697f94493d986c14f269f7f5fa 2023-04-05 18:56:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:18,827 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:18,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:18,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:18,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:18,827 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:18,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:18,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:18,830 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,833 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:18,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:18,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:18,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:18,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:18,835 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,836 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:18,836 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:18,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/b5d3d2625b7647758cf2aebd5badb8ac 2023-04-05 18:56:18,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:18,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:18,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/0fb3cfdaf9ca43efbe8752d8e147377c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0fb3cfdaf9ca43efbe8752d8e147377c 2023-04-05 18:56:18,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0fb3cfdaf9ca43efbe8752d8e147377c, entries=400, sequenceid=658, filesize=59.8 K 2023-04-05 18:56:18,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/4c1af5697f94493d986c14f269f7f5fa as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4c1af5697f94493d986c14f269f7f5fa 2023-04-05 18:56:18,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4c1af5697f94493d986c14f269f7f5fa, entries=150, sequenceid=658, filesize=11.9 K 2023-04-05 18:56:18,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/b5d3d2625b7647758cf2aebd5badb8ac as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b5d3d2625b7647758cf2aebd5badb8ac 2023-04-05 18:56:18,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b5d3d2625b7647758cf2aebd5badb8ac, entries=150, sequenceid=658, filesize=11.9 K 2023-04-05 18:56:18,849 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 891ms, sequenceid=658, compaction requested=true 2023-04-05 18:56:18,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:18,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:18,849 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 18:56:18,849 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 18:56:18,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:18,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:18,851 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 244224 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 18:56:18,851 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/A is initiating minor compaction (all files) 2023-04-05 18:56:18,851 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/A in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,851 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60da56b565b34026a9ca7b94b8fbe88a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fdc76454546643748ad801eab6bfa9b7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/d59ed6963be346f09e604fc19ca40a96, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7bde8e4f3b4c4212852cc60dfdc74a10, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a91856386dd140a8b53747ca37da7e5a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f8e761340e1f4a4390c76cae88a7cd83, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/bf04fead67c7404093c1bfe5cf4b9962, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0fb3cfdaf9ca43efbe8752d8e147377c] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=238.5 K 2023-04-05 18:56:18,852 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98170 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 18:56:18,852 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:18,852 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,852 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 60da56b565b34026a9ca7b94b8fbe88a, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680720972567 2023-04-05 18:56:18,852 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/95fc341adb5840bfb46fdfbb41624ced, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/66ca9482712946179b5ca8bdcc4701f3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6503a6e49c074fed8482cdd67df46afd, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4114e60a99a9440180c91ce1e4348f1d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6f1db48b7fd54c828ba8ea4714eb8387, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a34491dd6639490d814d8e415e29628f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/735e27a5a4064896bdefb428ab8b351b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4c1af5697f94493d986c14f269f7f5fa] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=95.9 K 2023-04-05 18:56:18,852 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting fdc76454546643748ad801eab6bfa9b7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680720972690 2023-04-05 18:56:18,852 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 95fc341adb5840bfb46fdfbb41624ced, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680720972567 2023-04-05 18:56:18,852 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting d59ed6963be346f09e604fc19ca40a96, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680720973309 2023-04-05 18:56:18,852 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 66ca9482712946179b5ca8bdcc4701f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680720972690 2023-04-05 18:56:18,853 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7bde8e4f3b4c4212852cc60dfdc74a10, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1680720974444 2023-04-05 18:56:18,853 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 6503a6e49c074fed8482cdd67df46afd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680720973309 2023-04-05 18:56:18,853 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting a91856386dd140a8b53747ca37da7e5a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1680720975559 2023-04-05 18:56:18,853 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4114e60a99a9440180c91ce1e4348f1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1680720974444 2023-04-05 18:56:18,853 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting f8e761340e1f4a4390c76cae88a7cd83, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680720976192 2023-04-05 18:56:18,853 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 6f1db48b7fd54c828ba8ea4714eb8387, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1680720975559 2023-04-05 18:56:18,853 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting bf04fead67c7404093c1bfe5cf4b9962, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1680720977313 2023-04-05 18:56:18,853 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a34491dd6639490d814d8e415e29628f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680720976192 2023-04-05 18:56:18,854 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 0fb3cfdaf9ca43efbe8752d8e147377c, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680720977942 2023-04-05 18:56:18,854 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 735e27a5a4064896bdefb428ab8b351b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1680720977313 2023-04-05 18:56:18,854 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4c1af5697f94493d986c14f269f7f5fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680720977945 2023-04-05 18:56:18,865 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#604 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:18,867 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#A#compaction#605 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:18,874 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/452f6fb3db574240b48ab2444b9a363e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/452f6fb3db574240b48ab2444b9a363e 2023-04-05 18:56:18,880 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 452f6fb3db574240b48ab2444b9a363e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:18,880 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:18,880 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=8, startTime=1680720978849; duration=0sec 2023-04-05 18:56:18,880 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:18,880 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 18:56:18,884 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98170 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 18:56:18,885 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction (all files) 2023-04-05 18:56:18,885 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,885 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21ddc171957a4204a180e2cc45bd5eb5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c5b73a7161ba4844887b129966f96a6d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/67393ef8c2e944b992af5c2136dbcbaa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/924e5225e6654eae91f2a051a2f3f481, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3361f54289dd4646963b865fe7b8478f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b70554e2c92d4ff0adc6e1d196c3b155, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/38c5012e9e534ae493cfd487ea2833f2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b5d3d2625b7647758cf2aebd5badb8ac] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=95.9 K 2023-04-05 18:56:18,892 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 21ddc171957a4204a180e2cc45bd5eb5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680720972567 2023-04-05 18:56:18,892 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting c5b73a7161ba4844887b129966f96a6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680720972690 2023-04-05 18:56:18,893 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 67393ef8c2e944b992af5c2136dbcbaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680720973309 2023-04-05 18:56:18,893 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 924e5225e6654eae91f2a051a2f3f481, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1680720974444 2023-04-05 18:56:18,893 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 3361f54289dd4646963b865fe7b8478f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1680720975559 2023-04-05 18:56:18,893 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting b70554e2c92d4ff0adc6e1d196c3b155, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680720976192 2023-04-05 18:56:18,894 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 38c5012e9e534ae493cfd487ea2833f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=647, earliestPutTs=1680720977313 2023-04-05 18:56:18,894 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting b5d3d2625b7647758cf2aebd5badb8ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680720977945 2023-04-05 18:56:18,895 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/b9afd51f8f724b178b20ba0ddc93d763 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b9afd51f8f724b178b20ba0ddc93d763 2023-04-05 18:56:18,902 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/A of 7a7f1bea453fa17f7cd1e6eff9f4a686 into b9afd51f8f724b178b20ba0ddc93d763(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:18,902 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:18,902 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/A, priority=8, startTime=1680720978849; duration=0sec 2023-04-05 18:56:18,902 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:18,906 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#606 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:18,913 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/8b1f3e920a7547baa8f1119ff7770d35 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8b1f3e920a7547baa8f1119ff7770d35 2023-04-05 18:56:18,916 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 8b1f3e920a7547baa8f1119ff7770d35(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:18,917 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:18,917 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=8, startTime=1680720978849; duration=0sec 2023-04-05 18:56:18,917 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:18,936 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:18,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:18,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:18,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:18,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:18,938 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:18,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:18,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:18,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,941 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,941 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:18,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,942 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:18,942 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:18,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:18,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:18,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:18,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:18,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:18,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:18,943 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:18,943 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:18,944 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-05 18:56:18,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:18,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:18,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:18,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:18,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:18,944 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:18,957 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304058bd718cc2edf4ca382b13611dfe98857 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058bd718cc2edf4ca382b13611dfe98857 2023-04-05 18:56:19,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:19,123 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:19,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721039131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721039131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721039131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721039134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721039134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721039236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721039239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721039239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,365 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=695, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/a8935967058c4e7db1bca6b8b7cb952b 2023-04-05 18:56:19,374 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/1b433a4aa271492e92c6f9c88a899697 2023-04-05 18:56:19,388 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/6cd91c78020446c4b56d12a0d2c201f5 2023-04-05 18:56:19,391 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/a8935967058c4e7db1bca6b8b7cb952b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a8935967058c4e7db1bca6b8b7cb952b 2023-04-05 18:56:19,396 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a8935967058c4e7db1bca6b8b7cb952b, entries=150, sequenceid=695, filesize=25.4 K 2023-04-05 18:56:19,397 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/1b433a4aa271492e92c6f9c88a899697 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1b433a4aa271492e92c6f9c88a899697 2023-04-05 18:56:19,401 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1b433a4aa271492e92c6f9c88a899697, entries=150, sequenceid=695, filesize=11.9 K 2023-04-05 18:56:19,402 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/6cd91c78020446c4b56d12a0d2c201f5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6cd91c78020446c4b56d12a0d2c201f5 2023-04-05 18:56:19,431 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6cd91c78020446c4b56d12a0d2c201f5, entries=150, sequenceid=695, filesize=11.9 K 2023-04-05 18:56:19,432 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 488ms, sequenceid=695, compaction requested=false 2023-04-05 18:56:19,432 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:19,432 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:19,433 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,436 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:19,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@140ee793[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:19,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:19,439 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:19,440 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 18:56:19,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:19,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:19,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:19,440 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:19,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:19,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:19,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:19,442 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:19,443 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:19,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:19,444 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:19,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:19,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:19,444 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:19,444 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:19,448 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,449 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,449 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:19,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,451 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,454 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:19,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:19,454 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:19,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:19,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:19,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:19,455 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:19,455 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:19,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:19,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721039464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721039468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721039472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,555 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:19,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:19,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:19,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:19,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:19,559 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:19,559 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:19,559 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:19,559 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:19,560 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,560 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,561 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,561 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:19,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:19,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,562 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:19,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:19,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:19,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:19,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:19,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:19,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:19,563 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:19,563 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:19,563 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:19,564 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:19,564 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:19,564 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:19,564 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:19,564 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,564 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:19,564 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:19,566 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:19,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c4608c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:19,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:19,568 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721039569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,570 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:19,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:19,570 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:19,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,570 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:19,571 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:19,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:19,572 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:19,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721039573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,575 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:19,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:19,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:19,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:19,575 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:19,575 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:19,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721039575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,577 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,579 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:19,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:19,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,582 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:19,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:19,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:19,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:19,582 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:19,582 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:19,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,582 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:19,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:19,682 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:19,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:19,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:19,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:19,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:19,686 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:19,686 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:19,686 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:19,686 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:19,686 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,686 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,688 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:19,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:19,689 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:19,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:19,691 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:19,691 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:19,691 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:19,691 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:19,692 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,692 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:19,693 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,693 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,693 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:19,700 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:19,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74654865[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:19,705 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:19,705 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,706 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:19,706 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,706 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:19,711 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:19,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:19,715 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:19,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:19,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:19,715 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:19,716 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:19,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,717 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,718 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,720 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:19,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:19,723 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,724 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:19,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:19,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:19,724 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:19,724 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:19,724 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,724 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:19,724 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,724 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,724 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,725 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:19,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721039771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721039777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:19,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721039777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,824 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:19,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:19,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:19,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:19,827 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:19,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:19,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:19,827 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:19,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,828 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,829 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:19,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:19,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,829 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:19,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:19,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:19,831 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:19,831 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:19,831 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:19,831 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:19,833 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,833 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:19,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1586d25[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:19,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:19,835 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,836 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:19,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:19,836 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:19,837 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,837 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:19,837 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:19,838 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:19,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:19,840 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:19,840 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:19,840 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:19,840 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:19,840 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:19,841 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,841 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:19,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,841 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,844 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,845 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:19,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:19,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:19,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:19,848 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:19,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,848 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:19,848 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:19,848 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,849 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,849 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,849 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:19,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:19,869 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304051db3ff72b18b47fba2d976ff8435d78d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051db3ff72b18b47fba2d976ff8435d78d 2023-04-05 18:56:19,874 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=707, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/dcace88cb5cc488787e3bb756be87d12 2023-04-05 18:56:19,949 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:19,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:19,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:19,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:19,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:19,952 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:19,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:19,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:19,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:19,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,953 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:19,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:19,953 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,953 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:19,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:19,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:19,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:19,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:19,955 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:19,957 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@628930e2[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:19,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:19,958 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,959 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:19,959 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,959 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:19,963 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:19,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:19,966 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:19,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:19,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:19,966 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:19,966 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:19,967 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,967 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:19,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,967 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,967 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:19,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,969 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:19,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:19,971 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:19,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:19,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:19,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:19,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:19,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:19,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:19,971 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:19,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:19,971 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:19,971 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:19,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,071 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:20,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:20,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:20,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:20,074 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:20,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:20,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:20,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,076 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,076 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721040077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:20,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:20,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:20,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:20,078 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:20,081 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,081 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d2fc3f[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:20,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:20,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:20,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721040082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:20,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721040082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,084 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:20,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:20,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:20,084 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:20,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:20,088 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:20,088 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:20,088 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:20,088 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:20,088 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:20,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,090 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,090 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,092 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,096 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,096 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,096 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:20,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:20,096 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:20,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,096 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:20,096 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:20,097 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:20,097 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:20,097 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:20,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721040141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:20,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721040143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,197 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:20,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:20,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:20,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:20,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:20,200 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:20,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:20,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:20,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,202 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,202 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,203 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:20,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:20,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:20,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:20,204 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,204 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:20,204 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:20,204 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:20,205 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,205 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,206 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,206 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:20,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27aaa500[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:20,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:20,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,208 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:20,209 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,209 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:20,213 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,214 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:20,216 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:20,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:20,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:20,216 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:20,216 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:20,217 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,217 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,217 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,218 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,219 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,221 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,221 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,221 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,221 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:20,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:20,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:20,221 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,221 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,221 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,222 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:20,222 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:20,221 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:20,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,221 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,222 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,283 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=707 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/a2d5e40dc504462da55edc8c75c629d5 2023-04-05 18:56:20,322 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:20,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:20,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:20,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:20,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:20,325 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:20,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:20,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:20,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,325 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,326 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,326 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:20,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:20,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:20,328 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,328 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:20,328 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:20,328 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:20,330 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:20,332 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:20,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d9c595[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:20,332 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,333 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,333 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:20,333 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:20,333 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:20,333 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,333 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:20,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:20,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:20,336 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:20,336 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:20,336 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:20,336 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:20,336 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:20,338 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,338 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:20,338 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,339 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:20,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:20,342 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:20,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,343 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,343 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,343 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:20,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:20,343 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:20,343 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:20,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,443 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:20,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:20,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:20,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:20,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:20,449 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:20,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:20,449 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:20,450 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,450 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,452 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:20,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:20,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:20,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:20,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:20,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:20,454 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,455 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:20,454 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:20,455 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:20,455 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:20,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:20,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:20,455 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,456 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,459 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,459 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:20,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4efcf7ec[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:20,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:20,461 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:20,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:20,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:20,462 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,462 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:20,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:20,465 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:20,467 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:20,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:20,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:20,467 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:20,467 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:20,469 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,469 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,469 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:20,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,470 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,471 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,478 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,478 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:20,478 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:20,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:20,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:20,478 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:20,478 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,478 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:20,478 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,478 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,479 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,578 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:20,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:20,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:20,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721040579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:20,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:20,584 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:20,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:20,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:20,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,584 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,586 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,586 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,587 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,587 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:20,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:20,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721040586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:20,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:20,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721040587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:20,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:20,596 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:20,596 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,596 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:20,608 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,608 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:20,609 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:20,609 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:20,611 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,612 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,612 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:20,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a1d908[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:20,613 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:20,613 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,616 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,616 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:20,616 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:20,616 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:20,616 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,616 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:20,617 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:20,617 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:20,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:20,621 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:20,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:20,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:20,621 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:20,621 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:20,623 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,623 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:20,623 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,623 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,625 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:20,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:20,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:20,628 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:20,629 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:20,628 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:20,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:20,628 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:20,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=707 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/5dfbb2787a244336a3e834de7abfa9ad 2023-04-05 18:56:20,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/dcace88cb5cc488787e3bb756be87d12 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/dcace88cb5cc488787e3bb756be87d12 2023-04-05 18:56:20,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/dcace88cb5cc488787e3bb756be87d12, entries=150, sequenceid=707, filesize=25.4 K 2023-04-05 18:56:20,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/a2d5e40dc504462da55edc8c75c629d5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a2d5e40dc504462da55edc8c75c629d5 2023-04-05 18:56:20,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a2d5e40dc504462da55edc8c75c629d5, entries=150, sequenceid=707, filesize=11.9 K 2023-04-05 18:56:20,714 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/5dfbb2787a244336a3e834de7abfa9ad as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/5dfbb2787a244336a3e834de7abfa9ad 2023-04-05 18:56:20,717 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/5dfbb2787a244336a3e834de7abfa9ad, entries=150, sequenceid=707, filesize=11.9 K 2023-04-05 18:56:20,717 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 1277ms, sequenceid=707, compaction requested=true 2023-04-05 18:56:20,718 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:20,718 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:20,718 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:20,718 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:20,718 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:20,718 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:20,719 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79058 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:20,719 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:20,719 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/A is initiating minor compaction (all files) 2023-04-05 18:56:20,719 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:20,719 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/A in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,719 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,719 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b9afd51f8f724b178b20ba0ddc93d763, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a8935967058c4e7db1bca6b8b7cb952b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/dcace88cb5cc488787e3bb756be87d12] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=77.2 K 2023-04-05 18:56:20,719 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/452f6fb3db574240b48ab2444b9a363e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1b433a4aa271492e92c6f9c88a899697, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a2d5e40dc504462da55edc8c75c629d5] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.6 K 2023-04-05 18:56:20,719 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting b9afd51f8f724b178b20ba0ddc93d763, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680720977945 2023-04-05 18:56:20,719 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 452f6fb3db574240b48ab2444b9a363e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680720977945 2023-04-05 18:56:20,720 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting a8935967058c4e7db1bca6b8b7cb952b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1680720978002 2023-04-05 18:56:20,720 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1b433a4aa271492e92c6f9c88a899697, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1680720978002 2023-04-05 18:56:20,720 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting dcace88cb5cc488787e3bb756be87d12, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680720979130 2023-04-05 18:56:20,720 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a2d5e40dc504462da55edc8c75c629d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680720979130 2023-04-05 18:56:20,729 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:20,730 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#614 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:20,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:20,734 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#A#compaction#613 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:20,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:20,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:20,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:20,737 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:20,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:20,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:20,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,737 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,739 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/872852dbc03b4f6b94adb41b7df26ca0 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/872852dbc03b4f6b94adb41b7df26ca0 2023-04-05 18:56:20,740 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:20,740 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:20,741 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:20,741 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:20,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:20,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:20,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:20,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:20,742 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:20,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:20,743 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 18:56:20,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:20,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:20,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:20,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:20,743 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:20,744 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:20,744 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 872852dbc03b4f6b94adb41b7df26ca0(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:20,744 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:20,744 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=13, startTime=1680720980718; duration=0sec 2023-04-05 18:56:20,744 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:20,744 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:20,746 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:20,746 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction (all files) 2023-04-05 18:56:20,746 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:20,746 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8b1f3e920a7547baa8f1119ff7770d35, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6cd91c78020446c4b56d12a0d2c201f5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/5dfbb2787a244336a3e834de7abfa9ad] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.6 K 2023-04-05 18:56:20,746 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 8b1f3e920a7547baa8f1119ff7770d35, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680720977945 2023-04-05 18:56:20,747 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 6cd91c78020446c4b56d12a0d2c201f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1680720978002 2023-04-05 18:56:20,747 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 5dfbb2787a244336a3e834de7abfa9ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680720979130 2023-04-05 18:56:20,753 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#616 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:20,758 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304059b4849b96365402f9e0f417472c44336 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059b4849b96365402f9e0f417472c44336 2023-04-05 18:56:20,763 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/abc45f23e07d441eb3b906ed16ef09b8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/abc45f23e07d441eb3b906ed16ef09b8 2023-04-05 18:56:20,767 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/A of 7a7f1bea453fa17f7cd1e6eff9f4a686 into abc45f23e07d441eb3b906ed16ef09b8(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:20,767 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:20,767 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/A, priority=13, startTime=1680720980718; duration=0sec 2023-04-05 18:56:20,767 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:20,770 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=727, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/7f9d73e21d44455fa2d0545a851de25c 2023-04-05 18:56:21,165 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/338e5153f49149fa9a56cf67c0e308f5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/338e5153f49149fa9a56cf67c0e308f5 2023-04-05 18:56:21,169 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 338e5153f49149fa9a56cf67c0e308f5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:21,169 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:21,169 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=13, startTime=1680720980718; duration=0sec 2023-04-05 18:56:21,169 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:21,196 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/d3712de8333a4b54b777e30f2c9908da 2023-04-05 18:56:21,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:21,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:21,605 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/9fab56269fef40eb89e367a100d8c597 2023-04-05 18:56:21,610 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/7f9d73e21d44455fa2d0545a851de25c as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7f9d73e21d44455fa2d0545a851de25c 2023-04-05 18:56:21,613 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7f9d73e21d44455fa2d0545a851de25c, entries=150, sequenceid=727, filesize=25.4 K 2023-04-05 18:56:21,614 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/d3712de8333a4b54b777e30f2c9908da as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3712de8333a4b54b777e30f2c9908da 2023-04-05 18:56:21,617 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3712de8333a4b54b777e30f2c9908da, entries=150, sequenceid=727, filesize=11.9 K 2023-04-05 18:56:21,618 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/9fab56269fef40eb89e367a100d8c597 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/9fab56269fef40eb89e367a100d8c597 2023-04-05 18:56:21,621 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/9fab56269fef40eb89e367a100d8c597, entries=150, sequenceid=727, filesize=11.9 K 2023-04-05 18:56:21,635 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=53.67 KB/54960 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 892ms, sequenceid=727, compaction requested=false 2023-04-05 18:56:21,635 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:21,636 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:21,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 18:56:21,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:21,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:21,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:21,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:21,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:21,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:21,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:21,638 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,638 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,638 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,638 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:21,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:21,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52d1eab9[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:21,644 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:21,644 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,649 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:21,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,650 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:21,652 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:21,656 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:21,656 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:21,656 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:21,656 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:21,656 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:21,658 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:21,660 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,660 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:21,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:21,661 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,662 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040501820c7d122449de83ca68f513c56f74 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040501820c7d122449de83ca68f513c56f74 2023-04-05 18:56:21,662 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,666 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,666 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,666 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:21,666 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,666 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:21,666 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:21,666 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,667 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,667 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:21,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:21,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:21,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:21,667 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,668 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:21,669 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:21,669 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=744, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/60c6905ffc894af386fbf4a8e6ad9bc3 2023-04-05 18:56:21,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:21,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721041666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:21,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721041668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:21,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721041670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,769 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:21,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:21,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:21,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:21,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721041772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:21,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721041772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:21,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721041774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:21,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:21,776 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:21,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:21,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:21,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:21,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,776 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,778 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,778 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:21,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:21,778 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,778 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:21,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:21,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:21,780 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:21,780 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:21,780 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:21,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:21,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:21,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:21,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:21,790 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:21,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790f781b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:21,791 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:21,792 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,794 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:21,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:21,794 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:21,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:21,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:21,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:21,799 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:21,799 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:21,799 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:21,799 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:21,799 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:21,801 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,801 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:21,801 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:21,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:21,801 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,803 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,805 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:21,805 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:21,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:21,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:21,805 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:21,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:21,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:21,806 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:21,806 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:21,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:21,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,806 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,906 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:21,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:21,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:21,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:21,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:21,909 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:21,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:21,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:21,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:21,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,914 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:21,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:21,914 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,915 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:21,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:21,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:21,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:21,916 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:21,916 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:21,916 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:21,916 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,918 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5065a341[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:21,919 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:21,920 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,920 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:21,921 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,921 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:21,923 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:21,926 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:21,926 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:21,926 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:21,926 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:21,926 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:21,927 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,927 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:21,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,927 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:21,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:21,928 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,929 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,931 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,931 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:21,931 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:21,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:21,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:21,931 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:21,931 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:21,931 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,931 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:21,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:21,932 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:21,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:21,932 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,932 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,932 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:21,932 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:21,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:21,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:21,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721041975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:21,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721041976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:21,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:21,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721041978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,032 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:22,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:22,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:22,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:22,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:22,035 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:22,035 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,035 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:22,035 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:22,035 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,035 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,036 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,036 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,037 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,037 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:22,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:22,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:22,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:22,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:22,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:22,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:22,038 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,038 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:22,038 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:22,038 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:22,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:22,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:22,038 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,040 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ea259dc[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:22,042 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:22,042 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,043 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:22,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,043 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:22,045 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:22,047 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:22,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:22,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:22,047 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:22,047 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:22,049 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,049 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,049 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,050 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,051 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,053 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,053 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,053 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,053 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:22,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:22,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,053 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,053 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:22,054 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,054 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:22,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,054 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,054 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:22,054 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,054 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/87765676dd1048d7acfe2f5ae6d358c9 2023-04-05 18:56:22,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/21c1f2d7b8744cccba9ce2945e26244e 2023-04-05 18:56:22,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/60c6905ffc894af386fbf4a8e6ad9bc3 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60c6905ffc894af386fbf4a8e6ad9bc3 2023-04-05 18:56:22,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60c6905ffc894af386fbf4a8e6ad9bc3, entries=150, sequenceid=744, filesize=25.4 K 2023-04-05 18:56:22,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/87765676dd1048d7acfe2f5ae6d358c9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/87765676dd1048d7acfe2f5ae6d358c9 2023-04-05 18:56:22,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/87765676dd1048d7acfe2f5ae6d358c9, entries=150, sequenceid=744, filesize=11.9 K 2023-04-05 18:56:22,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/21c1f2d7b8744cccba9ce2945e26244e as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21c1f2d7b8744cccba9ce2945e26244e 2023-04-05 18:56:22,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21c1f2d7b8744cccba9ce2945e26244e, entries=150, sequenceid=744, filesize=11.9 K 2023-04-05 18:56:22,106 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 469ms, sequenceid=744, compaction requested=true 2023-04-05 18:56:22,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:22,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:22,106 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:22,106 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:22,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:22,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:22,107 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:22,107 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79160 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:22,107 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:22,107 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/A is initiating minor compaction (all files) 2023-04-05 18:56:22,107 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,107 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/A in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,107 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/872852dbc03b4f6b94adb41b7df26ca0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3712de8333a4b54b777e30f2c9908da, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/87765676dd1048d7acfe2f5ae6d358c9] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.7 K 2023-04-05 18:56:22,107 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/abc45f23e07d441eb3b906ed16ef09b8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7f9d73e21d44455fa2d0545a851de25c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60c6905ffc894af386fbf4a8e6ad9bc3] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=77.3 K 2023-04-05 18:56:22,107 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 872852dbc03b4f6b94adb41b7df26ca0, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680720979130 2023-04-05 18:56:22,108 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting abc45f23e07d441eb3b906ed16ef09b8, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680720979130 2023-04-05 18:56:22,108 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting d3712de8333a4b54b777e30f2c9908da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1680720979461 2023-04-05 18:56:22,108 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 7f9d73e21d44455fa2d0545a851de25c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1680720979461 2023-04-05 18:56:22,108 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 87765676dd1048d7acfe2f5ae6d358c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680720981597 2023-04-05 18:56:22,108 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 60c6905ffc894af386fbf4a8e6ad9bc3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680720981597 2023-04-05 18:56:22,114 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#622 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:22,120 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#A#compaction#623 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:22,127 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/85771fad5af2493f843cacc27cf59eb9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/85771fad5af2493f843cacc27cf59eb9 2023-04-05 18:56:22,131 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 85771fad5af2493f843cacc27cf59eb9(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:22,131 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:22,131 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=13, startTime=1680720982106; duration=0sec 2023-04-05 18:56:22,132 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:22,132 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:22,133 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:22,133 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction (all files) 2023-04-05 18:56:22,133 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,133 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/338e5153f49149fa9a56cf67c0e308f5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/9fab56269fef40eb89e367a100d8c597, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21c1f2d7b8744cccba9ce2945e26244e] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.7 K 2023-04-05 18:56:22,133 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 338e5153f49149fa9a56cf67c0e308f5, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680720979130 2023-04-05 18:56:22,134 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 9fab56269fef40eb89e367a100d8c597, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1680720979461 2023-04-05 18:56:22,134 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 21c1f2d7b8744cccba9ce2945e26244e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680720981597 2023-04-05 18:56:22,141 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#624 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:22,148 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/1423bec7611a4d00922a61f8e6a73206 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/1423bec7611a4d00922a61f8e6a73206 2023-04-05 18:56:22,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:22,149 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 18:56:22,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:22,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:22,153 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 1423bec7611a4d00922a61f8e6a73206(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:22,153 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:22,153 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=13, startTime=1680720982106; duration=0sec 2023-04-05 18:56:22,153 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:22,154 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:22,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:22,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:22,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:22,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:22,158 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:22,158 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,158 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:22,158 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:22,158 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,159 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,160 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,160 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,161 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,161 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:22,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:22,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:22,162 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:22,162 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,162 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:22,162 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:22,162 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:22,162 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,162 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:22,162 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:22,162 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:22,165 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721042170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:22,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,173 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:22,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721042172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23fbe093[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:22,173 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,174 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:22,175 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:22,175 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:22,176 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405afa6eb039675412aaf4b274063aea67b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405afa6eb039675412aaf4b274063aea67b 2023-04-05 18:56:22,180 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,180 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:22,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:22,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,180 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,180 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:22,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:22,195 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:22,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:22,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:22,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:22,195 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:22,197 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,197 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,197 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,198 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=764, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/ac0699d7f499434f80bc2bc0802c6404 2023-04-05 18:56:22,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,201 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,207 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:22,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:22,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:22,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,208 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,208 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:22,208 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:22,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/18524cd9ba114a77a24ccf9a8841b91f 2023-04-05 18:56:22,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/95d1b9e2169d4580825bf07afa46d86f 2023-04-05 18:56:22,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/ac0699d7f499434f80bc2bc0802c6404 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/ac0699d7f499434f80bc2bc0802c6404 2023-04-05 18:56:22,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/ac0699d7f499434f80bc2bc0802c6404, entries=200, sequenceid=764, filesize=32.3 K 2023-04-05 18:56:22,228 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/18524cd9ba114a77a24ccf9a8841b91f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/18524cd9ba114a77a24ccf9a8841b91f 2023-04-05 18:56:22,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/18524cd9ba114a77a24ccf9a8841b91f, entries=150, sequenceid=764, filesize=11.9 K 2023-04-05 18:56:22,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/95d1b9e2169d4580825bf07afa46d86f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/95d1b9e2169d4580825bf07afa46d86f 2023-04-05 18:56:22,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/95d1b9e2169d4580825bf07afa46d86f, entries=150, sequenceid=764, filesize=11.9 K 2023-04-05 18:56:22,236 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 87ms, sequenceid=764, compaction requested=false 2023-04-05 18:56:22,236 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:22,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:22,275 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 18:56:22,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:22,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:22,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:22,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:22,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:22,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:22,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721042289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721042290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721042291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721042291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721042292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,308 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:22,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:22,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:22,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:22,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:22,311 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:22,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:22,311 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:22,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,312 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,313 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:22,316 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:22,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:22,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:22,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:22,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:22,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:22,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:22,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:22,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:22,318 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:22,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:22,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:22,318 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:22,325 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,325 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,326 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,326 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:22,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21368b57[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:22,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:22,327 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:22,333 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:22,333 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:22,333 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,333 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:22,334 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:22,334 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,334 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:22,334 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:22,338 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:22,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:22,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:22,338 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:22,338 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:22,342 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,343 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,344 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,346 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,346 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,346 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,346 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,346 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,346 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,346 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,346 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:22,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:22,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:22,347 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:22,347 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:22,348 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,348 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,348 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:22,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721042393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721042393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721042396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721042396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721042396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,447 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:22,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:22,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:22,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:22,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:22,452 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:22,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:22,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:22,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,452 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,455 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,455 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,455 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,455 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:22,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:22,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:22,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:22,460 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,460 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:22,460 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:22,460 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:22,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,462 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,462 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c345b9b[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:22,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:22,463 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,464 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:22,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,464 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,466 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:22,468 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:22,468 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:22,468 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:22,468 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:22,468 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:22,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,471 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,471 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,472 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,475 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,475 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,475 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:22,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:22,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,475 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,475 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:22,475 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:22,475 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,475 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,476 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,476 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,476 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,576 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:22,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:22,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:22,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:22,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:22,578 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:22,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:22,578 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:22,579 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,579 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,580 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,580 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,581 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:22,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:22,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:22,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:22,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:22,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:22,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:22,584 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,584 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:22,584 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:22,584 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:22,585 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:22,586 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,586 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30b48578[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:22,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:22,588 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,590 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,590 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:22,590 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:22,590 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:22,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,590 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:22,591 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,591 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,591 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:22,594 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:22,594 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:22,594 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:22,594 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:22,594 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:22,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,597 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,597 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,600 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721042598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721042599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,603 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:22,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:22,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721042599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721042599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,603 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:22,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721042600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,604 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:22,603 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:22,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,604 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,688 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040593ad9c99c6464c0484c41ffeb9565c15 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040593ad9c99c6464c0484c41ffeb9565c15 2023-04-05 18:56:22,697 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=780, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/f28f8439f5b148d9b1a2b5beb70b9b16 2023-04-05 18:56:22,704 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:22,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:22,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:22,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:22,718 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:22,718 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,718 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:22,718 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:22,728 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,728 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,730 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,730 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,731 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,731 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:22,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:22,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:22,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:22,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:22,747 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,747 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:22,747 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:22,747 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:22,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:22,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:22,747 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,749 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:22,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cb71485[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:22,750 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:22,750 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,751 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,752 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:22,752 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:22,752 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:22,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,752 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:22,770 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:22,786 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:22,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:22,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:22,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:22,786 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:22,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,788 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,789 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,792 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,794 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,794 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,794 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,795 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:22,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:22,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,795 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,795 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:22,795 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:22,895 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:22,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:22,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:22,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721042903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721042905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721042905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721042905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:22,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:22,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:22,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721042905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,908 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:22,908 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,908 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:22,908 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:22,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,909 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,910 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,910 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,911 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,911 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:22,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:22,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:22,921 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,921 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:22,921 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:22,921 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:22,946 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,946 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,947 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,947 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@594e4277[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:22,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:22,949 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:22,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:22,951 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:22,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:22,954 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:22,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:22,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:22,954 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:22,954 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:22,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:22,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,957 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:22,959 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:22,961 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/384cfc903aff410a8f7711839387265a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/384cfc903aff410a8f7711839387265a 2023-04-05 18:56:22,962 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,962 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,963 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:22,963 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:22,963 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:22,963 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,963 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:22,963 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:22,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:22,963 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:22,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:22,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:22,963 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:22,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:22,963 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:22,963 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:22,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:22,965 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/A of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 384cfc903aff410a8f7711839387265a(size=26.5 K), total size for store is 58.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:22,966 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:22,966 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/A, priority=13, startTime=1680720982106; duration=0sec 2023-04-05 18:56:22,966 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:23,063 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:23,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:23,072 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:23,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:23,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:23,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:23,073 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,073 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,074 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:23,074 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:23,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:23,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,075 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:23,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:23,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:23,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:23,076 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,076 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:23,077 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:23,077 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:23,080 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,080 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:23,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27583246[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:23,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:23,084 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,085 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,086 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:23,086 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:23,086 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:23,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,086 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:23,093 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:23,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:23,099 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:23,099 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:23,099 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:23,099 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:23,099 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:23,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,101 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:23,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,102 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,109 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,112 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:23,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:23,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:23,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:23,112 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:23,112 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:23,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:23,113 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:23,113 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:23,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:23,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/628798d690744234a1de9b5c811bfdbe 2023-04-05 18:56:23,213 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:23,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:23,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:23,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:23,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:23,220 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:23,221 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:23,221 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:23,221 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:23,221 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,221 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,222 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:23,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:23,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,222 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:23,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:23,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:23,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:23,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:23,224 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:23,226 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d3ec90[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:23,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:23,228 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,229 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:23,230 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,230 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:23,231 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,231 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,231 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:23,232 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:23,234 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:23,234 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:23,234 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:23,234 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:23,234 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:23,235 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,235 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,235 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:23,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,236 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,237 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:23,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:23,242 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:23,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:23,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,242 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:23,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:23,243 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:23,243 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:23,243 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:23,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:23,343 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:23,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:23,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:23,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:23,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:23,347 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:23,347 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:23,347 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:23,347 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:23,348 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,348 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,349 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:23,349 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,351 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,351 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:23,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:23,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:23,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:23,353 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,353 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:23,353 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:23,353 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,354 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:23,354 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:23,354 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,354 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:23,354 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,356 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:23,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4734c553[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:23,358 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:23,358 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,359 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:23,359 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,359 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:23,362 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:23,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,362 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:23,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:23,364 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:23,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:23,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:23,364 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:23,364 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:23,366 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,366 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:23,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,366 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,366 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,367 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:23,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:23,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:23,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,369 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:23,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:23,369 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:23,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:23,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,370 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:23,370 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:23,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:23,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:23,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721043410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:23,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721043410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:23,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721043411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:23,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721043411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:23,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721043412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,470 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:23,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:23,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:23,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:23,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:23,473 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:23,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:23,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:23,473 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:23,474 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,474 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,475 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:23,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:23,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,475 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:23,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:23,476 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:23,476 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:23,476 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:23,477 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:23,477 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:23,477 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,477 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:23,477 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:23,477 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,477 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:23,477 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:23,477 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,477 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:23,477 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:23,480 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,480 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,480 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,480 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc27a38[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:23,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:23,481 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,482 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,482 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:23,483 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:23,483 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:23,483 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,483 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:23,484 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,484 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:23,484 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:23,484 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:23,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:23,486 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:23,486 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:23,486 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:23,486 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:23,486 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:23,488 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,488 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,488 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:23,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:23,488 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,490 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,493 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,493 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,493 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:23,493 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:23,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:23,493 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:23,493 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,493 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:23,494 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,494 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:23,494 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:23,494 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:23,494 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:23,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:23,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:23,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/ecc9c8d6d8204bcbb78c66b561a381e1 2023-04-05 18:56:23,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/f28f8439f5b148d9b1a2b5beb70b9b16 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f28f8439f5b148d9b1a2b5beb70b9b16 2023-04-05 18:56:23,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f28f8439f5b148d9b1a2b5beb70b9b16, entries=250, sequenceid=780, filesize=39.2 K 2023-04-05 18:56:23,555 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/628798d690744234a1de9b5c811bfdbe as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/628798d690744234a1de9b5c811bfdbe 2023-04-05 18:56:23,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/628798d690744234a1de9b5c811bfdbe, entries=150, sequenceid=780, filesize=11.9 K 2023-04-05 18:56:23,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/ecc9c8d6d8204bcbb78c66b561a381e1 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/ecc9c8d6d8204bcbb78c66b561a381e1 2023-04-05 18:56:23,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/ecc9c8d6d8204bcbb78c66b561a381e1, entries=150, sequenceid=780, filesize=11.9 K 2023-04-05 18:56:23,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 1290ms, sequenceid=780, compaction requested=true 2023-04-05 18:56:23,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:23,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:23,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:23,565 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:23,565 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:23,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:23,566 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 100386 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:23,566 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/A is initiating minor compaction (all files) 2023-04-05 18:56:23,566 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/A in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,566 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/384cfc903aff410a8f7711839387265a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/ac0699d7f499434f80bc2bc0802c6404, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f28f8439f5b148d9b1a2b5beb70b9b16] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=98.0 K 2023-04-05 18:56:23,567 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:23,567 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:23,567 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,567 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/85771fad5af2493f843cacc27cf59eb9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/18524cd9ba114a77a24ccf9a8841b91f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/628798d690744234a1de9b5c811bfdbe] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.8 K 2023-04-05 18:56:23,567 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 384cfc903aff410a8f7711839387265a, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680720981597 2023-04-05 18:56:23,567 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 85771fad5af2493f843cacc27cf59eb9, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680720981597 2023-04-05 18:56:23,567 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting ac0699d7f499434f80bc2bc0802c6404, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1680720981665 2023-04-05 18:56:23,567 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 18524cd9ba114a77a24ccf9a8841b91f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1680720981667 2023-04-05 18:56:23,568 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting f28f8439f5b148d9b1a2b5beb70b9b16, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680720982163 2023-04-05 18:56:23,568 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 628798d690744234a1de9b5c811bfdbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680720982169 2023-04-05 18:56:23,574 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#632 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:23,577 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#A#compaction#631 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:23,594 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:23,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:23,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:23,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:23,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:23,597 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:23,597 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:23,597 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:23,597 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:23,597 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,597 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,598 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:23,598 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:23,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:23,599 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:23,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:23,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:23,600 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:23,600 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:23,600 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-05 18:56:23,601 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:23,601 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:23,601 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:23,601 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:23,601 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:23,601 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:23,618 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304053bfd1bb7d6f644a2b0b9bb6b6eec2b00 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053bfd1bb7d6f644a2b0b9bb6b6eec2b00 2023-04-05 18:56:23,633 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=798, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/1183f50de1904b5c98fbbe6c811c70a8 2023-04-05 18:56:23,649 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=798 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/4baf33803e2849539c81c22fa4f7d49f 2023-04-05 18:56:23,991 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/ae6de19bad824e31b3438ad27b3badaa as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ae6de19bad824e31b3438ad27b3badaa 2023-04-05 18:56:24,000 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into ae6de19bad824e31b3438ad27b3badaa(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:24,000 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:24,000 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=13, startTime=1680720983565; duration=0sec 2023-04-05 18:56:24,001 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:24,001 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:24,002 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:24,002 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction (all files) 2023-04-05 18:56:24,002 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,002 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/1423bec7611a4d00922a61f8e6a73206, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/95d1b9e2169d4580825bf07afa46d86f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/ecc9c8d6d8204bcbb78c66b561a381e1] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.8 K 2023-04-05 18:56:24,002 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 1423bec7611a4d00922a61f8e6a73206, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680720981597 2023-04-05 18:56:24,003 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 95d1b9e2169d4580825bf07afa46d86f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1680720981667 2023-04-05 18:56:24,003 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ecc9c8d6d8204bcbb78c66b561a381e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680720982169 2023-04-05 18:56:24,027 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/f6b973f9799441afbdc12dcc641e06e5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f6b973f9799441afbdc12dcc641e06e5 2023-04-05 18:56:24,030 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#636 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:24,032 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/A of 7a7f1bea453fa17f7cd1e6eff9f4a686 into f6b973f9799441afbdc12dcc641e06e5(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:24,032 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:24,032 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/A, priority=13, startTime=1680720983565; duration=0sec 2023-04-05 18:56:24,032 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:24,059 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/4531a7a724c34a5cb10d8c8393609d8b as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/4531a7a724c34a5cb10d8c8393609d8b 2023-04-05 18:56:24,063 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 4531a7a724c34a5cb10d8c8393609d8b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:24,063 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:24,064 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=13, startTime=1680720983565; duration=0sec 2023-04-05 18:56:24,064 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:24,077 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=798 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/a6b770f961154190971b28954cd18b86 2023-04-05 18:56:24,082 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/1183f50de1904b5c98fbbe6c811c70a8 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/1183f50de1904b5c98fbbe6c811c70a8 2023-04-05 18:56:24,091 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/1183f50de1904b5c98fbbe6c811c70a8, entries=150, sequenceid=798, filesize=25.4 K 2023-04-05 18:56:24,092 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/4baf33803e2849539c81c22fa4f7d49f as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4baf33803e2849539c81c22fa4f7d49f 2023-04-05 18:56:24,100 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4baf33803e2849539c81c22fa4f7d49f, entries=150, sequenceid=798, filesize=11.9 K 2023-04-05 18:56:24,101 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/a6b770f961154190971b28954cd18b86 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a6b770f961154190971b28954cd18b86 2023-04-05 18:56:24,105 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a6b770f961154190971b28954cd18b86, entries=150, sequenceid=798, filesize=11.9 K 2023-04-05 18:56:24,106 INFO [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 506ms, sequenceid=798, compaction requested=false 2023-04-05 18:56:24,106 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:24,106 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:24,108 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,108 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:24,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aeebfe4[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:24,120 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:24,120 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,122 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,122 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:24,122 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:24,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,122 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:24,122 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:24,124 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,124 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:24,124 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,124 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:24,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:24,127 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:24,127 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:24,127 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:24,128 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:24,128 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:24,129 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,129 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,129 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,130 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,132 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,142 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,142 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,142 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:24,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:24,142 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,142 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,143 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,143 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,143 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,143 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,143 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:24,143 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,143 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:24,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,243 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:24,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:24,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:24,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:24,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:24,256 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:24,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:24,256 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:24,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,257 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,259 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,259 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,260 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,260 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:24,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:24,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:24,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:24,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:24,268 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:24,268 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,269 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:24,269 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:24,269 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:24,271 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,271 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,271 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,272 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:24,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26f57545[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:24,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:24,273 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,274 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:24,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:24,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:24,276 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:24,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:24,276 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:24,277 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:24,277 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:24,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,279 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,280 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,281 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:24,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:24,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,284 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,284 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:24,284 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:24,284 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:24,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,384 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:24,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:24,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:24,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:24,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:24,388 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:24,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:24,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:24,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,389 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,390 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,390 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,391 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,391 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:24,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:24,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:24,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:24,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:24,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:24,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,392 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:24,400 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,400 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:24,400 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:24,400 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:24,403 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,403 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,403 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:24,406 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@478a3295[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:24,408 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:24,408 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,410 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:24,411 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:24,411 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:24,411 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,411 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,412 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,412 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:24,414 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:24,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:24,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:24,414 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:24,415 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:24,416 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,416 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,416 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,416 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,418 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,419 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,419 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,420 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,420 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,419 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,420 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,420 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,420 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,420 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:24,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:24,420 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,420 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:24,420 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:24,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:24,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:24,437 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 18:56:24,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:24,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:24,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:24,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:24,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:24,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:24,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720984451, max timestamp=1680720984456Num uniques 859; 2023-04-05 18:56:24,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:56:24,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720984451, max timestamp=1680720984456Num uniques 859; 2023-04-05 18:56:24,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:56:24,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680720984451, max timestamp=1680720984456Num uniques 859; 2023-04-05 18:56:24,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 18:56:24,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721044470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721044470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721044475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721044476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721044476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,520 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:24,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:24,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:24,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:24,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:24,524 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:24,524 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,524 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:24,524 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:24,524 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,524 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,526 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,526 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,527 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,527 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:24,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:24,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:24,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:24,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:24,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,529 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:24,534 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,534 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:24,534 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:24,534 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:24,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:24,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:24,535 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,536 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38e693e6[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:24,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:24,539 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,540 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,541 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:24,541 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:24,541 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:24,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,541 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,542 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:24,545 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:24,545 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:24,545 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:24,545 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:24,545 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:24,548 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,548 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,548 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,549 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,550 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,553 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,553 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,553 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,553 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,553 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,553 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:24,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:24,553 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,553 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,553 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,553 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,553 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:24,553 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:24,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721044577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721044577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721044579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721044581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721044581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,653 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:24,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:24,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:24,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:24,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:24,657 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:24,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:24,657 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:24,658 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,658 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,663 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,663 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:24,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:24,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:24,665 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,665 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:24,665 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:24,665 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:24,666 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,666 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47081a81[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:24,668 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:24,668 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,669 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,669 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:24,669 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:24,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,669 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:24,669 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:24,671 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,671 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:24,671 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,671 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:24,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:24,673 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:24,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:24,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:24,673 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:24,673 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:24,675 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,675 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,675 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,676 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,677 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,679 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,679 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,679 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,679 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,679 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,679 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,679 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:24,679 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,679 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:24,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:24,679 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,679 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:24,679 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:24,780 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:24,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:24,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721044779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:24,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:24,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:24,783 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:24,783 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,783 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:24,783 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:24,784 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,784 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,785 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,785 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721044782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721044783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,786 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:24,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:24,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:24,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721044784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:24,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:24,787 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:24,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721044785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:24,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:24,788 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:24,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:24,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:24,788 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:24,789 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,789 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:24,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ada0078[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:24,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:24,794 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,796 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:24,796 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,796 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:24,797 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:24,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:24,802 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:24,802 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:24,802 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:24,802 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:24,802 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:24,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,804 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,805 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,807 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,810 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:24,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:24,810 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,810 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:24,810 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:24,851 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304058f9180dee0f94af58b9767e3b392c85b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058f9180dee0f94af58b9767e3b392c85b 2023-04-05 18:56:24,910 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:24,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:24,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:24,916 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:24,916 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,916 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:24,916 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:24,916 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,917 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,918 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,918 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,918 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,918 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:24,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:24,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:24,920 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:24,920 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:24,920 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:24,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:24,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:24,934 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:24,936 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:24,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cd28d2c[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:24,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:24,939 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,940 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:24,941 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,941 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:24,943 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:24,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:24,948 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:24,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:24,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:24,948 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:24,948 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:24,949 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,949 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:24,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,950 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:24,950 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,952 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:24,955 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:24,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:24,956 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:24,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:24,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:24,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:24,956 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:24,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:24,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:24,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:24,956 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:24,956 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:24,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:25,057 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:25,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:25,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:25,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:25,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:25,060 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:25,060 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:25,060 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:25,060 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:25,060 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,060 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,062 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:25,062 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:25,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,063 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:25,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:25,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:25,064 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:25,064 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:25,064 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:25,064 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:25,069 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,069 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:25,069 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,069 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:25,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:25,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a75cf3[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:25,071 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:25,071 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,072 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:25,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,072 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:25,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:25,072 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:25,075 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:25,077 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:25,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:25,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:25,077 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:25,077 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:25,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,079 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:25,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,080 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,081 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,082 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:25,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:25,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:25,083 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:25,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:25,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:25,083 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:25,083 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:25,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:25,083 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:25,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721045084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:25,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721045086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:25,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33489] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721045087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:25,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721045088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:25,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33489] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721045090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,183 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:25,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:25,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:25,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:25,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:25,187 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:25,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:25,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:25,187 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:25,188 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,188 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,189 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:25,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:25,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,189 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:25,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:25,190 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:25,190 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:25,191 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:25,191 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:25,193 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57fc9874[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:25,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:25,195 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,196 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:25,196 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,196 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:25,197 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:25,199 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:25,199 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:25,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:25,200 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:25,200 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:25,205 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,205 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:25,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:25,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,206 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,207 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,210 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:25,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:25,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:25,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:25,210 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:25,211 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:25,211 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:25,211 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:25,211 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,211 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,211 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,211 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:25,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:25,255 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=813, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/9fafdb8fe63846d985594653029915a2 2023-04-05 18:56:25,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/2b8bc700285240e9888ba9bd43970b29 2023-04-05 18:56:25,311 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:25,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:25,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-05 18:56:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 18:56:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 18:56:25,314 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 18:56:25,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:25,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 18:56:25,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 18:56:25,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,314 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,315 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:25,315 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:25,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,316 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 18:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 18:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 18:56:25,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 18:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 18:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 18:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 18:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 18:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 18:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. started... 2023-04-05 18:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 18:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. as already flushing 2023-04-05 18:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,33489,1680720816595)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 18:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 18:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 18:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 18:56:25,319 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:25,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,319 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 18:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33489,1680720816595' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 18:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dbf9b33[Count = 0] remaining members to acquire global barrier 2023-04-05 18:56:25,320 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 18:56:25,320 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,322 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 18:56:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 18:56:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33489,1680720816595' in zk 2023-04-05 18:56:25,322 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,322 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 18:56:25,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 18:56:25,323 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:25,323 DEBUG [member: 'jenkins-hbase4.apache.org,33489,1680720816595' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 18:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33489,1680720816595': 2023-04-05 18:56:25,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33489,1680720816595' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 18:56:25,326 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 18:56:25,326 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 18:56:25,326 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 18:56:25,326 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 18:56:25,326 INFO [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 18:56:25,327 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,327 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:25,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,327 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 18:56:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 18:56:25,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:25,328 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 18:56:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 18:56:25,329 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 18:56:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 18:56:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 18:56:25,331 DEBUG [(jenkins-hbase4.apache.org,39855,1680720815401)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 18:56:25,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 18:56:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 18:56:25,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 18:56:25,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 18:56:25,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 18:56:25,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 18:56:25,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 18:56:25,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 18:56:25,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,331 DEBUG [Thread-3889] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 18:56:25,331 DEBUG [Thread-3889] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 18:56:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 18:56:25,331 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 18:56:25,332 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 18:56:25,390 DEBUG [Thread-3894] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32bf16cd to 127.0.0.1:49755 2023-04-05 18:56:25,390 DEBUG [Thread-3894] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:25,390 DEBUG [Thread-3896] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28164190 to 127.0.0.1:49755 2023-04-05 18:56:25,390 DEBUG [Thread-3896] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:25,391 DEBUG [Thread-3890] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b59bdcf to 127.0.0.1:49755 2023-04-05 18:56:25,392 DEBUG [Thread-3890] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:25,394 DEBUG [Thread-3898] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x308b3f39 to 127.0.0.1:49755 2023-04-05 18:56:25,394 DEBUG [Thread-3898] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:25,394 DEBUG [Thread-3892] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:49755 2023-04-05 18:56:25,395 DEBUG [Thread-3892] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:25,432 DEBUG [Thread-3889] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 18:56:25,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 18:56:25,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56720 deadline: 1680721045588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:25,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56746 deadline: 1680721045591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:25,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56722 deadline: 1680721045592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:25,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56696 deadline: 1680721045593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 18:56:25,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56730 deadline: 1680721045594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:25,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/83bd349d430445c48c2a57058bb9507a 2023-04-05 18:56:25,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/9fafdb8fe63846d985594653029915a2 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/9fafdb8fe63846d985594653029915a2 2023-04-05 18:56:25,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/9fafdb8fe63846d985594653029915a2, entries=350, sequenceid=813, filesize=52.9 K 2023-04-05 18:56:25,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/2b8bc700285240e9888ba9bd43970b29 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/2b8bc700285240e9888ba9bd43970b29 2023-04-05 18:56:25,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/2b8bc700285240e9888ba9bd43970b29, entries=150, sequenceid=813, filesize=11.9 K 2023-04-05 18:56:25,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/83bd349d430445c48c2a57058bb9507a as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/83bd349d430445c48c2a57058bb9507a 2023-04-05 18:56:25,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/83bd349d430445c48c2a57058bb9507a, entries=150, sequenceid=813, filesize=11.9 K 2023-04-05 18:56:25,688 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 1251ms, sequenceid=813, compaction requested=true 2023-04-05 18:56:25,688 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:25,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:25,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:25,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:25,689 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:25,689 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:25,690 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:25,690 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107532 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:25,690 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/B is initiating minor compaction (all files) 2023-04-05 18:56:25,690 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/A is initiating minor compaction (all files) 2023-04-05 18:56:25,690 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/B in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:25,690 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/A in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:25,690 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ae6de19bad824e31b3438ad27b3badaa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4baf33803e2849539c81c22fa4f7d49f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/2b8bc700285240e9888ba9bd43970b29] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.9 K 2023-04-05 18:56:25,690 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f6b973f9799441afbdc12dcc641e06e5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/1183f50de1904b5c98fbbe6c811c70a8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/9fafdb8fe63846d985594653029915a2] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=105.0 K 2023-04-05 18:56:25,690 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting ae6de19bad824e31b3438ad27b3badaa, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680720982169 2023-04-05 18:56:25,690 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting f6b973f9799441afbdc12dcc641e06e5, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680720982169 2023-04-05 18:56:25,691 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4baf33803e2849539c81c22fa4f7d49f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=798, earliestPutTs=1680720982288 2023-04-05 18:56:25,691 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 1183f50de1904b5c98fbbe6c811c70a8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=798, earliestPutTs=1680720982288 2023-04-05 18:56:25,691 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 2b8bc700285240e9888ba9bd43970b29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1680720984428 2023-04-05 18:56:25,691 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] compactions.Compactor(207): Compacting 9fafdb8fe63846d985594653029915a2, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1680720984428 2023-04-05 18:56:25,698 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#B#compaction#641 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:25,700 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#A#compaction#640 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:26,105 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/92a92e162fd6481aa1be6653ee89c305 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/92a92e162fd6481aa1be6653ee89c305 2023-04-05 18:56:26,108 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/B of 7a7f1bea453fa17f7cd1e6eff9f4a686 into 92a92e162fd6481aa1be6653ee89c305(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:26,108 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:26,109 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/B, priority=13, startTime=1680720985689; duration=0sec 2023-04-05 18:56:26,109 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 18:56:26,109 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 18:56:26,109 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 18:56:26,110 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1912): 7a7f1bea453fa17f7cd1e6eff9f4a686/C is initiating minor compaction (all files) 2023-04-05 18:56:26,110 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7a7f1bea453fa17f7cd1e6eff9f4a686/C in TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:26,110 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/4531a7a724c34a5cb10d8c8393609d8b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a6b770f961154190971b28954cd18b86, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/83bd349d430445c48c2a57058bb9507a] into tmpdir=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp, totalSize=36.9 K 2023-04-05 18:56:26,110 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 4531a7a724c34a5cb10d8c8393609d8b, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680720982169 2023-04-05 18:56:26,110 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting a6b770f961154190971b28954cd18b86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=798, earliestPutTs=1680720982288 2023-04-05 18:56:26,110 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] compactions.Compactor(207): Compacting 83bd349d430445c48c2a57058bb9507a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1680720984428 2023-04-05 18:56:26,113 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/d081a8aba5344144a254780f577dc293 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/d081a8aba5344144a254780f577dc293 2023-04-05 18:56:26,115 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] throttle.PressureAwareThroughputController(145): 7a7f1bea453fa17f7cd1e6eff9f4a686#C#compaction#642 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 18:56:26,116 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/A of 7a7f1bea453fa17f7cd1e6eff9f4a686 into d081a8aba5344144a254780f577dc293(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:26,116 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:26,116 INFO [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/A, priority=13, startTime=1680720985689; duration=0sec 2023-04-05 18:56:26,116 DEBUG [RS:0;jenkins-hbase4:33489-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:26,522 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/f40081b524854b4da7ba457fd9b7b829 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/f40081b524854b4da7ba457fd9b7b829 2023-04-05 18:56:26,525 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7a7f1bea453fa17f7cd1e6eff9f4a686/C of 7a7f1bea453fa17f7cd1e6eff9f4a686 into f40081b524854b4da7ba457fd9b7b829(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 18:56:26,525 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:26,525 INFO [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686., storeName=7a7f1bea453fa17f7cd1e6eff9f4a686/C, priority=13, startTime=1680720985689; duration=0sec 2023-04-05 18:56:26,525 DEBUG [RS:0;jenkins-hbase4:33489-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 18:56:26,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33489] regionserver.HRegion(9158): Flush requested on 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:26,596 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-05 18:56:26,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:26,596 DEBUG [Thread-3879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:49755 2023-04-05 18:56:26,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:26,596 DEBUG [Thread-3879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:26,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:26,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:26,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:26,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 18:56:26,598 DEBUG [Thread-3885] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06799455 to 127.0.0.1:49755 2023-04-05 18:56:26,598 DEBUG [Thread-3881] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:49755 2023-04-05 18:56:26,598 DEBUG [Thread-3885] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:26,598 DEBUG [Thread-3881] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:26,602 DEBUG [Thread-3883] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:49755 2023-04-05 18:56:26,603 DEBUG [Thread-3883] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:26,604 DEBUG [Thread-3887] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:49755 2023-04-05 18:56:26,604 DEBUG [Thread-3887] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:26,604 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 18:56:26,604 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-04-05 18:56:26,604 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 157 2023-04-05 18:56:26,604 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 148 2023-04-05 18:56:26,604 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 146 2023-04-05 18:56:26,605 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(394): wrote 153 2023-04-05 18:56:26,605 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 18:56:26,605 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 4342 2023-04-05 18:56:26,605 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 4293 2023-04-05 18:56:26,605 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 4314 2023-04-05 18:56:26,605 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 4366 2023-04-05 18:56:26,605 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(398): read 4333 2023-04-05 18:56:26,605 INFO [Listener at localhost/43705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 18:56:26,605 INFO [Listener at localhost/43705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 18:56:26,605 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f06dd50 to 127.0.0.1:49755 2023-04-05 18:56:26,605 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:26,605 INFO [Listener at localhost/43705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 18:56:26,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-05 18:56:26,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:26,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 18:56:26,608 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720986608"}]},"ts":"1680720986608"} 2023-04-05 18:56:26,609 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 18:56:26,613 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 18:56:26,614 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, UNASSIGN}] 2023-04-05 18:56:26,615 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, UNASSIGN 2023-04-05 18:56:26,616 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=7a7f1bea453fa17f7cd1e6eff9f4a686, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:26,616 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720986616"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680720986616"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680720986616"}]},"ts":"1680720986616"} 2023-04-05 18:56:26,617 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595}] 2023-04-05 18:56:26,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 18:56:26,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:26,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7a7f1bea453fa17f7cd1e6eff9f4a686, disabling compactions & flushes 2023-04-05 18:56:26,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:26,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 18:56:27,008 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405c9bf355ef1824f3399e785048cde5d04 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c9bf355ef1824f3399e785048cde5d04 2023-04-05 18:56:27,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 18:56:27,413 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=853, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/0675d5dbc29e4f7992a79d04d64c6ea7 2023-04-05 18:56:27,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/188e3a31a219427aa79f723472c9b279 2023-04-05 18:56:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 18:56:27,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/0a8f0192b08e40fba07ade89738cd246 2023-04-05 18:56:27,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/0675d5dbc29e4f7992a79d04d64c6ea7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0675d5dbc29e4f7992a79d04d64c6ea7 2023-04-05 18:56:27,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0675d5dbc29e4f7992a79d04d64c6ea7, entries=150, sequenceid=853, filesize=25.4 K 2023-04-05 18:56:27,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/188e3a31a219427aa79f723472c9b279 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/188e3a31a219427aa79f723472c9b279 2023-04-05 18:56:27,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/188e3a31a219427aa79f723472c9b279, entries=150, sequenceid=853, filesize=11.9 K 2023-04-05 18:56:27,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/0a8f0192b08e40fba07ade89738cd246 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/0a8f0192b08e40fba07ade89738cd246 2023-04-05 18:56:27,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/0a8f0192b08e40fba07ade89738cd246, entries=150, sequenceid=853, filesize=11.9 K 2023-04-05 18:56:27,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=26.84 KB/27480 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 1245ms, sequenceid=853, compaction requested=false 2023-04-05 18:56:27,841 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:27,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:27,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:27,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. after waiting 0 ms 2023-04-05 18:56:27,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:27,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7a7f1bea453fa17f7cd1e6eff9f4a686 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-05 18:56:27,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=A 2023-04-05 18:56:27,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:27,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=B 2023-04-05 18:56:27,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:27,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7a7f1bea453fa17f7cd1e6eff9f4a686, store=C 2023-04-05 18:56:27,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 18:56:28,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405a99a655bb857417d9f209a374dbb08bd to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a99a655bb857417d9f209a374dbb08bd 2023-04-05 18:56:28,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=860, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/11057e2a4d904262ab200c5a20e9bca5 2023-04-05 18:56:28,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=860 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/30f7f6cabf334865a6d671fa78674286 2023-04-05 18:56:28,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=860 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/a22022712bd04378b2bcbde1a5bc2f61 2023-04-05 18:56:28,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/A/11057e2a4d904262ab200c5a20e9bca5 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/11057e2a4d904262ab200c5a20e9bca5 2023-04-05 18:56:28,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/11057e2a4d904262ab200c5a20e9bca5, entries=100, sequenceid=860, filesize=18.6 K 2023-04-05 18:56:28,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/B/30f7f6cabf334865a6d671fa78674286 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/30f7f6cabf334865a6d671fa78674286 2023-04-05 18:56:28,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/30f7f6cabf334865a6d671fa78674286, entries=100, sequenceid=860, filesize=9.5 K 2023-04-05 18:56:28,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/.tmp/C/a22022712bd04378b2bcbde1a5bc2f61 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a22022712bd04378b2bcbde1a5bc2f61 2023-04-05 18:56:28,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a22022712bd04378b2bcbde1a5bc2f61, entries=100, sequenceid=860, filesize=9.5 K 2023-04-05 18:56:28,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 7a7f1bea453fa17f7cd1e6eff9f4a686 in 856ms, sequenceid=860, compaction requested=true 2023-04-05 18:56:28,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 18:56:28,721 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/2535a52f1fea42c68a54e67bcde09a6f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/eb75e8d442a644a7895b5f9a29475149, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/98404499ba7a4aefa6fa05eb44cb1d59, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b869e2a0bbc0444fb698c522f483b9c8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/261071167e9f45c59df19d1ba7db9ecc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0bf8315dce624d53847da0b98f100466, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/5b2cec50de40481ba327f2f43bc7ba35, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/80aa58fd6e9945059b659ff33e04e0a8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/56fed31f7090462c9290e1b8ba279c60, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71505000faa54e0ba03f7d4773d23b3e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c2640d28af664ffd8dee2b6aa6ade32e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fd1257cddc1a4a178626e19fcd6d9cc9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/e4ec620d4e6145558ae8512eace0b0de, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/09efbb489451490eb0b08a53197bf224, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/05362fe9a750426f866d5a0ba35fc4ba, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c5b9e1c60db64f9f944df80939f1bdf4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/61549a07255c4399b1fdb7c55d5093f0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71739b1eb82a44288aae3b8932324e29, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c64a78de0e0841ed9cc398ea9dc7f7af, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60da56b565b34026a9ca7b94b8fbe88a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fdc76454546643748ad801eab6bfa9b7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/d59ed6963be346f09e604fc19ca40a96, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7bde8e4f3b4c4212852cc60dfdc74a10, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a91856386dd140a8b53747ca37da7e5a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f8e761340e1f4a4390c76cae88a7cd83, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/bf04fead67c7404093c1bfe5cf4b9962, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0fb3cfdaf9ca43efbe8752d8e147377c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b9afd51f8f724b178b20ba0ddc93d763, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a8935967058c4e7db1bca6b8b7cb952b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/abc45f23e07d441eb3b906ed16ef09b8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/dcace88cb5cc488787e3bb756be87d12, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7f9d73e21d44455fa2d0545a851de25c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/384cfc903aff410a8f7711839387265a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60c6905ffc894af386fbf4a8e6ad9bc3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/ac0699d7f499434f80bc2bc0802c6404, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f28f8439f5b148d9b1a2b5beb70b9b16, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f6b973f9799441afbdc12dcc641e06e5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/1183f50de1904b5c98fbbe6c811c70a8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/9fafdb8fe63846d985594653029915a2] to archive 2023-04-05 18:56:28,722 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:56:28,723 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/2535a52f1fea42c68a54e67bcde09a6f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/2535a52f1fea42c68a54e67bcde09a6f 2023-04-05 18:56:28,724 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/eb75e8d442a644a7895b5f9a29475149 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/eb75e8d442a644a7895b5f9a29475149 2023-04-05 18:56:28,725 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/98404499ba7a4aefa6fa05eb44cb1d59 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/98404499ba7a4aefa6fa05eb44cb1d59 2023-04-05 18:56:28,726 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b869e2a0bbc0444fb698c522f483b9c8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b869e2a0bbc0444fb698c522f483b9c8 2023-04-05 18:56:28,727 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/261071167e9f45c59df19d1ba7db9ecc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/261071167e9f45c59df19d1ba7db9ecc 2023-04-05 18:56:28,728 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0bf8315dce624d53847da0b98f100466 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0bf8315dce624d53847da0b98f100466 2023-04-05 18:56:28,729 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/5b2cec50de40481ba327f2f43bc7ba35 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/5b2cec50de40481ba327f2f43bc7ba35 2023-04-05 18:56:28,730 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/80aa58fd6e9945059b659ff33e04e0a8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/80aa58fd6e9945059b659ff33e04e0a8 2023-04-05 18:56:28,730 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/56fed31f7090462c9290e1b8ba279c60 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/56fed31f7090462c9290e1b8ba279c60 2023-04-05 18:56:28,731 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71505000faa54e0ba03f7d4773d23b3e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71505000faa54e0ba03f7d4773d23b3e 2023-04-05 18:56:28,732 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c2640d28af664ffd8dee2b6aa6ade32e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c2640d28af664ffd8dee2b6aa6ade32e 2023-04-05 18:56:28,733 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fd1257cddc1a4a178626e19fcd6d9cc9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fd1257cddc1a4a178626e19fcd6d9cc9 2023-04-05 18:56:28,734 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/e4ec620d4e6145558ae8512eace0b0de to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/e4ec620d4e6145558ae8512eace0b0de 2023-04-05 18:56:28,734 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/09efbb489451490eb0b08a53197bf224 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/09efbb489451490eb0b08a53197bf224 2023-04-05 18:56:28,735 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/05362fe9a750426f866d5a0ba35fc4ba to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/05362fe9a750426f866d5a0ba35fc4ba 2023-04-05 18:56:28,736 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c5b9e1c60db64f9f944df80939f1bdf4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c5b9e1c60db64f9f944df80939f1bdf4 2023-04-05 18:56:28,736 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/61549a07255c4399b1fdb7c55d5093f0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/61549a07255c4399b1fdb7c55d5093f0 2023-04-05 18:56:28,737 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71739b1eb82a44288aae3b8932324e29 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/71739b1eb82a44288aae3b8932324e29 2023-04-05 18:56:28,738 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c64a78de0e0841ed9cc398ea9dc7f7af to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/c64a78de0e0841ed9cc398ea9dc7f7af 2023-04-05 18:56:28,739 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60da56b565b34026a9ca7b94b8fbe88a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60da56b565b34026a9ca7b94b8fbe88a 2023-04-05 18:56:28,740 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fdc76454546643748ad801eab6bfa9b7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/fdc76454546643748ad801eab6bfa9b7 2023-04-05 18:56:28,741 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/d59ed6963be346f09e604fc19ca40a96 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/d59ed6963be346f09e604fc19ca40a96 2023-04-05 18:56:28,741 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7bde8e4f3b4c4212852cc60dfdc74a10 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7bde8e4f3b4c4212852cc60dfdc74a10 2023-04-05 18:56:28,742 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a91856386dd140a8b53747ca37da7e5a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a91856386dd140a8b53747ca37da7e5a 2023-04-05 18:56:28,743 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f8e761340e1f4a4390c76cae88a7cd83 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f8e761340e1f4a4390c76cae88a7cd83 2023-04-05 18:56:28,744 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/bf04fead67c7404093c1bfe5cf4b9962 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/bf04fead67c7404093c1bfe5cf4b9962 2023-04-05 18:56:28,745 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0fb3cfdaf9ca43efbe8752d8e147377c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0fb3cfdaf9ca43efbe8752d8e147377c 2023-04-05 18:56:28,746 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b9afd51f8f724b178b20ba0ddc93d763 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/b9afd51f8f724b178b20ba0ddc93d763 2023-04-05 18:56:28,747 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a8935967058c4e7db1bca6b8b7cb952b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/a8935967058c4e7db1bca6b8b7cb952b 2023-04-05 18:56:28,748 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/abc45f23e07d441eb3b906ed16ef09b8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/abc45f23e07d441eb3b906ed16ef09b8 2023-04-05 18:56:28,749 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/dcace88cb5cc488787e3bb756be87d12 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/dcace88cb5cc488787e3bb756be87d12 2023-04-05 18:56:28,749 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7f9d73e21d44455fa2d0545a851de25c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/7f9d73e21d44455fa2d0545a851de25c 2023-04-05 18:56:28,750 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/384cfc903aff410a8f7711839387265a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/384cfc903aff410a8f7711839387265a 2023-04-05 18:56:28,751 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60c6905ffc894af386fbf4a8e6ad9bc3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/60c6905ffc894af386fbf4a8e6ad9bc3 2023-04-05 18:56:28,752 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/ac0699d7f499434f80bc2bc0802c6404 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/ac0699d7f499434f80bc2bc0802c6404 2023-04-05 18:56:28,753 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f28f8439f5b148d9b1a2b5beb70b9b16 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f28f8439f5b148d9b1a2b5beb70b9b16 2023-04-05 18:56:28,754 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f6b973f9799441afbdc12dcc641e06e5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/f6b973f9799441afbdc12dcc641e06e5 2023-04-05 18:56:28,755 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/1183f50de1904b5c98fbbe6c811c70a8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/1183f50de1904b5c98fbbe6c811c70a8 2023-04-05 18:56:28,756 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/9fafdb8fe63846d985594653029915a2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/9fafdb8fe63846d985594653029915a2 2023-04-05 18:56:28,759 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4de162e188b54af9a07c2a4736afaf00, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d68de7fc83aa45e886a7afaef7245545, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/afed465fa75449cbb4a13dfada3b85fc, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ee2da8ae8b6a4c37898b41a3444b5db4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/04810b45ddeb4dada8874fb0ce76cfab, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d4defe034fb54b43b7edb10459e86bd8, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/8a14479e3d1f477cbb4a3f6ce9bb842f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/42ec2bc370ff411581baf87608048867, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4aeeea7e2e14470088905f3613844eca, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/b6dc9bfbb8bb4e60aad1dd56673be5f9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/3597d99588d04002bf0cd432f90e6e55, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3732ff3eb3941bc900c6a3ee5f642e4, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1c531d22eca449b0b07d5722f5fbc889, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/9d701842e2724f5f96105b9bbbcf0d5b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/322a469e1a4f4a15b4b4269889028549, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/cab4f353ca0a44da810ad1ddb72b1adb, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/54d311c7981843bf879bb296eb69b20e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/c8524ecb747c4ea9893cffd3b1a9470f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/933b438eaeb54d22be1f3a47f0b09e1b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/95fc341adb5840bfb46fdfbb41624ced, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/e0bbb341488442b996bcd48c157306af, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/66ca9482712946179b5ca8bdcc4701f3, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6503a6e49c074fed8482cdd67df46afd, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4114e60a99a9440180c91ce1e4348f1d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6f1db48b7fd54c828ba8ea4714eb8387, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a34491dd6639490d814d8e415e29628f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/735e27a5a4064896bdefb428ab8b351b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/452f6fb3db574240b48ab2444b9a363e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4c1af5697f94493d986c14f269f7f5fa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1b433a4aa271492e92c6f9c88a899697, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/872852dbc03b4f6b94adb41b7df26ca0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a2d5e40dc504462da55edc8c75c629d5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3712de8333a4b54b777e30f2c9908da, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/85771fad5af2493f843cacc27cf59eb9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/87765676dd1048d7acfe2f5ae6d358c9, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/18524cd9ba114a77a24ccf9a8841b91f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ae6de19bad824e31b3438ad27b3badaa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/628798d690744234a1de9b5c811bfdbe, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4baf33803e2849539c81c22fa4f7d49f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/2b8bc700285240e9888ba9bd43970b29] to archive 2023-04-05 18:56:28,760 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:56:28,761 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4de162e188b54af9a07c2a4736afaf00 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4de162e188b54af9a07c2a4736afaf00 2023-04-05 18:56:28,762 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d68de7fc83aa45e886a7afaef7245545 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d68de7fc83aa45e886a7afaef7245545 2023-04-05 18:56:28,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/afed465fa75449cbb4a13dfada3b85fc to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/afed465fa75449cbb4a13dfada3b85fc 2023-04-05 18:56:28,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ee2da8ae8b6a4c37898b41a3444b5db4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ee2da8ae8b6a4c37898b41a3444b5db4 2023-04-05 18:56:28,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/04810b45ddeb4dada8874fb0ce76cfab to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/04810b45ddeb4dada8874fb0ce76cfab 2023-04-05 18:56:28,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d4defe034fb54b43b7edb10459e86bd8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d4defe034fb54b43b7edb10459e86bd8 2023-04-05 18:56:28,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/8a14479e3d1f477cbb4a3f6ce9bb842f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/8a14479e3d1f477cbb4a3f6ce9bb842f 2023-04-05 18:56:28,767 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/42ec2bc370ff411581baf87608048867 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/42ec2bc370ff411581baf87608048867 2023-04-05 18:56:28,768 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4aeeea7e2e14470088905f3613844eca to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4aeeea7e2e14470088905f3613844eca 2023-04-05 18:56:28,769 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/b6dc9bfbb8bb4e60aad1dd56673be5f9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/b6dc9bfbb8bb4e60aad1dd56673be5f9 2023-04-05 18:56:28,769 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/3597d99588d04002bf0cd432f90e6e55 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/3597d99588d04002bf0cd432f90e6e55 2023-04-05 18:56:28,770 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3732ff3eb3941bc900c6a3ee5f642e4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3732ff3eb3941bc900c6a3ee5f642e4 2023-04-05 18:56:28,771 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1c531d22eca449b0b07d5722f5fbc889 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1c531d22eca449b0b07d5722f5fbc889 2023-04-05 18:56:28,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/9d701842e2724f5f96105b9bbbcf0d5b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/9d701842e2724f5f96105b9bbbcf0d5b 2023-04-05 18:56:28,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/322a469e1a4f4a15b4b4269889028549 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/322a469e1a4f4a15b4b4269889028549 2023-04-05 18:56:28,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/cab4f353ca0a44da810ad1ddb72b1adb to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/cab4f353ca0a44da810ad1ddb72b1adb 2023-04-05 18:56:28,774 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/54d311c7981843bf879bb296eb69b20e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/54d311c7981843bf879bb296eb69b20e 2023-04-05 18:56:28,775 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/c8524ecb747c4ea9893cffd3b1a9470f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/c8524ecb747c4ea9893cffd3b1a9470f 2023-04-05 18:56:28,775 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/933b438eaeb54d22be1f3a47f0b09e1b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/933b438eaeb54d22be1f3a47f0b09e1b 2023-04-05 18:56:28,776 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/95fc341adb5840bfb46fdfbb41624ced to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/95fc341adb5840bfb46fdfbb41624ced 2023-04-05 18:56:28,777 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/e0bbb341488442b996bcd48c157306af to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/e0bbb341488442b996bcd48c157306af 2023-04-05 18:56:28,778 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/66ca9482712946179b5ca8bdcc4701f3 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/66ca9482712946179b5ca8bdcc4701f3 2023-04-05 18:56:28,779 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6503a6e49c074fed8482cdd67df46afd to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6503a6e49c074fed8482cdd67df46afd 2023-04-05 18:56:28,779 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4114e60a99a9440180c91ce1e4348f1d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4114e60a99a9440180c91ce1e4348f1d 2023-04-05 18:56:28,780 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6f1db48b7fd54c828ba8ea4714eb8387 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/6f1db48b7fd54c828ba8ea4714eb8387 2023-04-05 18:56:28,781 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a34491dd6639490d814d8e415e29628f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a34491dd6639490d814d8e415e29628f 2023-04-05 18:56:28,782 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/735e27a5a4064896bdefb428ab8b351b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/735e27a5a4064896bdefb428ab8b351b 2023-04-05 18:56:28,783 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/452f6fb3db574240b48ab2444b9a363e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/452f6fb3db574240b48ab2444b9a363e 2023-04-05 18:56:28,784 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4c1af5697f94493d986c14f269f7f5fa to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4c1af5697f94493d986c14f269f7f5fa 2023-04-05 18:56:28,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1b433a4aa271492e92c6f9c88a899697 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/1b433a4aa271492e92c6f9c88a899697 2023-04-05 18:56:28,786 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/872852dbc03b4f6b94adb41b7df26ca0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/872852dbc03b4f6b94adb41b7df26ca0 2023-04-05 18:56:28,787 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a2d5e40dc504462da55edc8c75c629d5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/a2d5e40dc504462da55edc8c75c629d5 2023-04-05 18:56:28,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3712de8333a4b54b777e30f2c9908da to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/d3712de8333a4b54b777e30f2c9908da 2023-04-05 18:56:28,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/85771fad5af2493f843cacc27cf59eb9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/85771fad5af2493f843cacc27cf59eb9 2023-04-05 18:56:28,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/87765676dd1048d7acfe2f5ae6d358c9 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/87765676dd1048d7acfe2f5ae6d358c9 2023-04-05 18:56:28,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/18524cd9ba114a77a24ccf9a8841b91f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/18524cd9ba114a77a24ccf9a8841b91f 2023-04-05 18:56:28,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ae6de19bad824e31b3438ad27b3badaa to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/ae6de19bad824e31b3438ad27b3badaa 2023-04-05 18:56:28,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/628798d690744234a1de9b5c811bfdbe to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/628798d690744234a1de9b5c811bfdbe 2023-04-05 18:56:28,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4baf33803e2849539c81c22fa4f7d49f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/4baf33803e2849539c81c22fa4f7d49f 2023-04-05 18:56:28,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/2b8bc700285240e9888ba9bd43970b29 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/2b8bc700285240e9888ba9bd43970b29 2023-04-05 18:56:28,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/bdd2ac88b5a34b07bb636c121cccd024, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/97c3bf19cc7d4dec9397ca7dca1f2d6f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/4dc89b2b93904cc4b534566e70807206, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/655bcaa5335640d0bf2ebcdc4c7f803f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/962da7e23444443fb4c23652bda4ac2f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8ca0687f5af9466f8c21ef6673ecd621, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/58b5dd1df457451387784798d8737539, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/f7c62e8bccad41709bd07263bac68d1b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a7ff1a88ba49410285e45bed0fc2675d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/e527745707ad406d87dd9937b15857ed, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/1eb2e363ca7e4777a973f4543fba68b5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/0aa9712a125c49c9a18c09a6a2edb8aa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c0dd86d9676e4bb99ba03287fc65552c, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fe161307cf6a4e6e9a845758183afc53, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3e8f4b7284c94a93b6deccafb4ec9cc7, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6fff6379679b4df3bba50fdcbd19491b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/37e118a448454d6cbffdc175d2d2798a, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/27a7fe3cec404d19af56e912626449d1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fc3e3d8abd56406492f1b3342a5e33f0, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21ddc171957a4204a180e2cc45bd5eb5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/63ef95ea1f614270a0a0c3fd521f8347, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c5b73a7161ba4844887b129966f96a6d, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/67393ef8c2e944b992af5c2136dbcbaa, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/924e5225e6654eae91f2a051a2f3f481, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3361f54289dd4646963b865fe7b8478f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b70554e2c92d4ff0adc6e1d196c3b155, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/38c5012e9e534ae493cfd487ea2833f2, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8b1f3e920a7547baa8f1119ff7770d35, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b5d3d2625b7647758cf2aebd5badb8ac, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6cd91c78020446c4b56d12a0d2c201f5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/338e5153f49149fa9a56cf67c0e308f5, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/5dfbb2787a244336a3e834de7abfa9ad, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/9fab56269fef40eb89e367a100d8c597, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/1423bec7611a4d00922a61f8e6a73206, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21c1f2d7b8744cccba9ce2945e26244e, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/95d1b9e2169d4580825bf07afa46d86f, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/4531a7a724c34a5cb10d8c8393609d8b, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/ecc9c8d6d8204bcbb78c66b561a381e1, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a6b770f961154190971b28954cd18b86, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/83bd349d430445c48c2a57058bb9507a] to archive 2023-04-05 18:56:28,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 18:56:28,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/bdd2ac88b5a34b07bb636c121cccd024 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/bdd2ac88b5a34b07bb636c121cccd024 2023-04-05 18:56:28,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/97c3bf19cc7d4dec9397ca7dca1f2d6f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/97c3bf19cc7d4dec9397ca7dca1f2d6f 2023-04-05 18:56:28,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/4dc89b2b93904cc4b534566e70807206 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/4dc89b2b93904cc4b534566e70807206 2023-04-05 18:56:28,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/655bcaa5335640d0bf2ebcdc4c7f803f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/655bcaa5335640d0bf2ebcdc4c7f803f 2023-04-05 18:56:28,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/962da7e23444443fb4c23652bda4ac2f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/962da7e23444443fb4c23652bda4ac2f 2023-04-05 18:56:28,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8ca0687f5af9466f8c21ef6673ecd621 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8ca0687f5af9466f8c21ef6673ecd621 2023-04-05 18:56:28,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/58b5dd1df457451387784798d8737539 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/58b5dd1df457451387784798d8737539 2023-04-05 18:56:28,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/f7c62e8bccad41709bd07263bac68d1b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/f7c62e8bccad41709bd07263bac68d1b 2023-04-05 18:56:28,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a7ff1a88ba49410285e45bed0fc2675d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a7ff1a88ba49410285e45bed0fc2675d 2023-04-05 18:56:28,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/e527745707ad406d87dd9937b15857ed to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/e527745707ad406d87dd9937b15857ed 2023-04-05 18:56:28,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/1eb2e363ca7e4777a973f4543fba68b5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/1eb2e363ca7e4777a973f4543fba68b5 2023-04-05 18:56:28,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/0aa9712a125c49c9a18c09a6a2edb8aa to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/0aa9712a125c49c9a18c09a6a2edb8aa 2023-04-05 18:56:28,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c0dd86d9676e4bb99ba03287fc65552c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c0dd86d9676e4bb99ba03287fc65552c 2023-04-05 18:56:28,809 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fe161307cf6a4e6e9a845758183afc53 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fe161307cf6a4e6e9a845758183afc53 2023-04-05 18:56:28,810 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3e8f4b7284c94a93b6deccafb4ec9cc7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3e8f4b7284c94a93b6deccafb4ec9cc7 2023-04-05 18:56:28,811 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6fff6379679b4df3bba50fdcbd19491b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6fff6379679b4df3bba50fdcbd19491b 2023-04-05 18:56:28,811 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/37e118a448454d6cbffdc175d2d2798a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/37e118a448454d6cbffdc175d2d2798a 2023-04-05 18:56:28,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/27a7fe3cec404d19af56e912626449d1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/27a7fe3cec404d19af56e912626449d1 2023-04-05 18:56:28,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fc3e3d8abd56406492f1b3342a5e33f0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/fc3e3d8abd56406492f1b3342a5e33f0 2023-04-05 18:56:28,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21ddc171957a4204a180e2cc45bd5eb5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21ddc171957a4204a180e2cc45bd5eb5 2023-04-05 18:56:28,814 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/63ef95ea1f614270a0a0c3fd521f8347 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/63ef95ea1f614270a0a0c3fd521f8347 2023-04-05 18:56:28,815 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c5b73a7161ba4844887b129966f96a6d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/c5b73a7161ba4844887b129966f96a6d 2023-04-05 18:56:28,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/67393ef8c2e944b992af5c2136dbcbaa to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/67393ef8c2e944b992af5c2136dbcbaa 2023-04-05 18:56:28,817 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/924e5225e6654eae91f2a051a2f3f481 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/924e5225e6654eae91f2a051a2f3f481 2023-04-05 18:56:28,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3361f54289dd4646963b865fe7b8478f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/3361f54289dd4646963b865fe7b8478f 2023-04-05 18:56:28,821 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b70554e2c92d4ff0adc6e1d196c3b155 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b70554e2c92d4ff0adc6e1d196c3b155 2023-04-05 18:56:28,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/38c5012e9e534ae493cfd487ea2833f2 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/38c5012e9e534ae493cfd487ea2833f2 2023-04-05 18:56:28,824 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8b1f3e920a7547baa8f1119ff7770d35 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/8b1f3e920a7547baa8f1119ff7770d35 2023-04-05 18:56:28,825 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b5d3d2625b7647758cf2aebd5badb8ac to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/b5d3d2625b7647758cf2aebd5badb8ac 2023-04-05 18:56:28,826 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6cd91c78020446c4b56d12a0d2c201f5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/6cd91c78020446c4b56d12a0d2c201f5 2023-04-05 18:56:28,827 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/338e5153f49149fa9a56cf67c0e308f5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/338e5153f49149fa9a56cf67c0e308f5 2023-04-05 18:56:28,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/5dfbb2787a244336a3e834de7abfa9ad to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/5dfbb2787a244336a3e834de7abfa9ad 2023-04-05 18:56:28,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/9fab56269fef40eb89e367a100d8c597 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/9fab56269fef40eb89e367a100d8c597 2023-04-05 18:56:28,830 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/1423bec7611a4d00922a61f8e6a73206 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/1423bec7611a4d00922a61f8e6a73206 2023-04-05 18:56:28,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21c1f2d7b8744cccba9ce2945e26244e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/21c1f2d7b8744cccba9ce2945e26244e 2023-04-05 18:56:28,832 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/95d1b9e2169d4580825bf07afa46d86f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/95d1b9e2169d4580825bf07afa46d86f 2023-04-05 18:56:28,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/4531a7a724c34a5cb10d8c8393609d8b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/4531a7a724c34a5cb10d8c8393609d8b 2023-04-05 18:56:28,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/ecc9c8d6d8204bcbb78c66b561a381e1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/ecc9c8d6d8204bcbb78c66b561a381e1 2023-04-05 18:56:28,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a6b770f961154190971b28954cd18b86 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a6b770f961154190971b28954cd18b86 2023-04-05 18:56:28,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/83bd349d430445c48c2a57058bb9507a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/83bd349d430445c48c2a57058bb9507a 2023-04-05 18:56:28,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/recovered.edits/863.seqid, newMaxSeqId=863, maxSeqId=4 2023-04-05 18:56:28,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686. 2023-04-05 18:56:28,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7a7f1bea453fa17f7cd1e6eff9f4a686: 2023-04-05 18:56:28,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:28,844 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=7a7f1bea453fa17f7cd1e6eff9f4a686, regionState=CLOSED 2023-04-05 18:56:28,844 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680720988844"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680720988844"}]},"ts":"1680720988844"} 2023-04-05 18:56:28,846 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-05 18:56:28,846 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 7a7f1bea453fa17f7cd1e6eff9f4a686, server=jenkins-hbase4.apache.org,33489,1680720816595 in 2.2270 sec 2023-04-05 18:56:28,847 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-05 18:56:28,847 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7a7f1bea453fa17f7cd1e6eff9f4a686, UNASSIGN in 2.2320 sec 2023-04-05 18:56:28,848 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680720988848"}]},"ts":"1680720988848"} 2023-04-05 18:56:28,848 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 18:56:28,850 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 18:56:28,851 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2440 sec 2023-04-05 18:56:30,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 18:56:30,721 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-05 18:56:30,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-05 18:56:30,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:30,724 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:30,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-05 18:56:30,724 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:30,729 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:30,730 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C, FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/recovered.edits] 2023-04-05 18:56:30,732 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0675d5dbc29e4f7992a79d04d64c6ea7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/0675d5dbc29e4f7992a79d04d64c6ea7 2023-04-05 18:56:30,733 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/11057e2a4d904262ab200c5a20e9bca5 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/11057e2a4d904262ab200c5a20e9bca5 2023-04-05 18:56:30,734 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/d081a8aba5344144a254780f577dc293 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/A/d081a8aba5344144a254780f577dc293 2023-04-05 18:56:30,736 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/188e3a31a219427aa79f723472c9b279 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/188e3a31a219427aa79f723472c9b279 2023-04-05 18:56:30,737 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/30f7f6cabf334865a6d671fa78674286 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/30f7f6cabf334865a6d671fa78674286 2023-04-05 18:56:30,738 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/92a92e162fd6481aa1be6653ee89c305 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/B/92a92e162fd6481aa1be6653ee89c305 2023-04-05 18:56:30,740 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/0a8f0192b08e40fba07ade89738cd246 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/0a8f0192b08e40fba07ade89738cd246 2023-04-05 18:56:30,741 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a22022712bd04378b2bcbde1a5bc2f61 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/a22022712bd04378b2bcbde1a5bc2f61 2023-04-05 18:56:30,742 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/f40081b524854b4da7ba457fd9b7b829 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/C/f40081b524854b4da7ba457fd9b7b829 2023-04-05 18:56:30,744 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/recovered.edits/863.seqid to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686/recovered.edits/863.seqid 2023-04-05 18:56:30,745 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/.tmp/data/default/TestAcidGuarantees/7a7f1bea453fa17f7cd1e6eff9f4a686 2023-04-05 18:56:30,745 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 18:56:30,745 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 18:56:30,746 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-05 18:56:30,749 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040501820c7d122449de83ca68f513c56f74 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040501820c7d122449de83ca68f513c56f74 2023-04-05 18:56:30,750 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040505c198cf13ff48c0ab9dd425278ad2cd to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040505c198cf13ff48c0ab9dd425278ad2cd 2023-04-05 18:56:30,751 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040505cbb268bb154027943a40babf1d3798 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040505cbb268bb154027943a40babf1d3798 2023-04-05 18:56:30,752 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051bbdf954addc48779a5821405848632c to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051bbdf954addc48779a5821405848632c 2023-04-05 18:56:30,753 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051db3ff72b18b47fba2d976ff8435d78d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051db3ff72b18b47fba2d976ff8435d78d 2023-04-05 18:56:30,755 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040539270aa38a9f4eabae7e7bf7e552334d to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040539270aa38a9f4eabae7e7bf7e552334d 2023-04-05 18:56:30,756 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053bfd1bb7d6f644a2b0b9bb6b6eec2b00 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053bfd1bb7d6f644a2b0b9bb6b6eec2b00 2023-04-05 18:56:30,757 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054e987f37d0fe45ff9c1ede7f233d0439 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054e987f37d0fe45ff9c1ede7f233d0439 2023-04-05 18:56:30,758 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040554ea51466ee8440cae5fc91c44aa0670 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040554ea51466ee8440cae5fc91c44aa0670 2023-04-05 18:56:30,759 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055d9a89af5d554965b86fd5b196f1a872 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055d9a89af5d554965b86fd5b196f1a872 2023-04-05 18:56:30,760 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405653dfc8045aa4df996102a982a4d39d7 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405653dfc8045aa4df996102a982a4d39d7 2023-04-05 18:56:30,761 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304057901226bd5374230a4d1ce6afca7abae to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304057901226bd5374230a4d1ce6afca7abae 2023-04-05 18:56:30,762 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040582cb096be40040639f66ab006ec7b356 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040582cb096be40040639f66ab006ec7b356 2023-04-05 18:56:30,763 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058bd718cc2edf4ca382b13611dfe98857 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058bd718cc2edf4ca382b13611dfe98857 2023-04-05 18:56:30,764 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058f9180dee0f94af58b9767e3b392c85b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058f9180dee0f94af58b9767e3b392c85b 2023-04-05 18:56:30,764 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040593ad9c99c6464c0484c41ffeb9565c15 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040593ad9c99c6464c0484c41ffeb9565c15 2023-04-05 18:56:30,765 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059744534d5569413b9f72b0473813590b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059744534d5569413b9f72b0473813590b 2023-04-05 18:56:30,766 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059b4849b96365402f9e0f417472c44336 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059b4849b96365402f9e0f417472c44336 2023-04-05 18:56:30,767 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059c0b461658fa4cb5b27e4ef8b6236107 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304059c0b461658fa4cb5b27e4ef8b6236107 2023-04-05 18:56:30,768 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a6ade5100bba432586b8e39a8417a146 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a6ade5100bba432586b8e39a8417a146 2023-04-05 18:56:30,769 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a99a655bb857417d9f209a374dbb08bd to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a99a655bb857417d9f209a374dbb08bd 2023-04-05 18:56:30,770 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405afa6eb039675412aaf4b274063aea67b to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405afa6eb039675412aaf4b274063aea67b 2023-04-05 18:56:30,771 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b6e6f8dfc18d422298a4c80d02bf86d1 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b6e6f8dfc18d422298a4c80d02bf86d1 2023-04-05 18:56:30,772 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b8ec0a046e594ac29607e54c44b175f0 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b8ec0a046e594ac29607e54c44b175f0 2023-04-05 18:56:30,772 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bf4cd97b04e94adcaad4d3585f4599e8 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bf4cd97b04e94adcaad4d3585f4599e8 2023-04-05 18:56:30,773 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c0d828ad70ab4fd58b8872abf483ce0a to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c0d828ad70ab4fd58b8872abf483ce0a 2023-04-05 18:56:30,774 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c9bf355ef1824f3399e785048cde5d04 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c9bf355ef1824f3399e785048cde5d04 2023-04-05 18:56:30,775 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ca6de1563ad94aeb8281eed2fa32f1d4 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ca6de1563ad94aeb8281eed2fa32f1d4 2023-04-05 18:56:30,776 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d5581b41f47a4bffa68ecd1cd8d9d35f to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d5581b41f47a4bffa68ecd1cd8d9d35f 2023-04-05 18:56:30,776 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405dbd2d4700af747c2a074c8a063163113 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405dbd2d4700af747c2a074c8a063163113 2023-04-05 18:56:30,777 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e416a1c1be2d4bf0a3214b06fb03229e to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e416a1c1be2d4bf0a3214b06fb03229e 2023-04-05 18:56:30,778 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e8a9707d09f84a7d9434cfc1144df671 to hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e8a9707d09f84a7d9434cfc1144df671 2023-04-05 18:56:30,779 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 18:56:30,780 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:30,782 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 18:56:30,783 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 18:56:30,784 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:30,784 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 18:56:30,784 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680720990784"}]},"ts":"9223372036854775807"} 2023-04-05 18:56:30,785 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 18:56:30,785 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7a7f1bea453fa17f7cd1e6eff9f4a686, NAME => 'TestAcidGuarantees,,1680720963623.7a7f1bea453fa17f7cd1e6eff9f4a686.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 18:56:30,785 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 18:56:30,785 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680720990785"}]},"ts":"9223372036854775807"} 2023-04-05 18:56:30,786 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 18:56:30,788 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 18:56:30,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 66 msec 2023-04-05 18:56:30,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39855] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-05 18:56:30,825 INFO [Listener at localhost/43705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-05 18:56:30,837 INFO [Listener at localhost/43705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 230), OpenFileDescriptor=574 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=358 (was 372), ProcessCount=169 (was 169), AvailableMemoryMB=1540 (was 1913) 2023-04-05 18:56:30,837 INFO [Listener at localhost/43705] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-05 18:56:30,837 INFO [Listener at localhost/43705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 18:56:30,838 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x643ba65c to 127.0.0.1:49755 2023-04-05 18:56:30,838 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:30,838 DEBUG [Listener at localhost/43705] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-05 18:56:30,838 DEBUG [Listener at localhost/43705] util.JVMClusterUtil(257): Found active master hash=748220392, stopped=false 2023-04-05 18:56:30,838 INFO [Listener at localhost/43705] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,39855,1680720815401 2023-04-05 18:56:30,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-05 18:56:30,840 INFO [Listener at localhost/43705] procedure2.ProcedureExecutor(629): Stopping 2023-04-05 18:56:30,841 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 18:56:30,840 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-05 18:56:30,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 18:56:30,841 DEBUG [Listener at localhost/43705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30327639 to 127.0.0.1:49755 2023-04-05 18:56:30,841 DEBUG [Listener at localhost/43705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:30,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 18:56:30,841 INFO [Listener at localhost/43705] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33489,1680720816595' ***** 2023-04-05 18:56:30,841 INFO [Listener at localhost/43705] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-05 18:56:30,842 INFO [RS:0;jenkins-hbase4:33489] regionserver.HeapMemoryManager(220): Stopping 2023-04-05 18:56:30,842 INFO [RS:0;jenkins-hbase4:33489] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-05 18:56:30,842 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-05 18:56:30,842 INFO [RS:0;jenkins-hbase4:33489] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-05 18:56:30,842 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(3304): Received CLOSE for 482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:56:30,843 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:30,843 DEBUG [RS:0;jenkins-hbase4:33489] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ecd2ede to 127.0.0.1:49755 2023-04-05 18:56:30,844 DEBUG [RS:0;jenkins-hbase4:33489] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:30,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 482b8a813b86c6c27617f295eb5f6e78, disabling compactions & flushes 2023-04-05 18:56:30,844 INFO [RS:0;jenkins-hbase4:33489] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-05 18:56:30,844 INFO [RS:0;jenkins-hbase4:33489] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-05 18:56:30,844 INFO [RS:0;jenkins-hbase4:33489] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-05 18:56:30,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:56:30,844 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-05 18:56:30,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:56:30,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. after waiting 0 ms 2023-04-05 18:56:30,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:56:30,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 482b8a813b86c6c27617f295eb5f6e78 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-05 18:56:30,844 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-05 18:56:30,844 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1479): Online Regions={482b8a813b86c6c27617f295eb5f6e78=hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78., 1588230740=hbase:meta,,1.1588230740} 2023-04-05 18:56:30,845 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-05 18:56:30,845 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-05 18:56:30,845 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-05 18:56:30,845 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-05 18:56:30,845 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-05 18:56:30,845 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-05 18:56:30,847 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1505): Waiting on 1588230740, 482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:56:30,909 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-05 18:56:31,047 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1505): Waiting on 1588230740, 482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:56:31,247 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1505): Waiting on 1588230740, 482b8a813b86c6c27617f295eb5f6e78 2023-04-05 18:56:31,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78/.tmp/info/ce8a4fdda3a9456ca717419247177416 2023-04-05 18:56:31,252 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/.tmp/info/59f18cbf905549afbf33e1becccc40d9 2023-04-05 18:56:31,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78/.tmp/info/ce8a4fdda3a9456ca717419247177416 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78/info/ce8a4fdda3a9456ca717419247177416 2023-04-05 18:56:31,256 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 59f18cbf905549afbf33e1becccc40d9 2023-04-05 18:56:31,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78/info/ce8a4fdda3a9456ca717419247177416, entries=2, sequenceid=6, filesize=4.8 K 2023-04-05 18:56:31,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 482b8a813b86c6c27617f295eb5f6e78 in 416ms, sequenceid=6, compaction requested=false 2023-04-05 18:56:31,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/namespace/482b8a813b86c6c27617f295eb5f6e78/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-05 18:56:31,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:56:31,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 482b8a813b86c6c27617f295eb5f6e78: 2023-04-05 18:56:31,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680720818777.482b8a813b86c6c27617f295eb5f6e78. 2023-04-05 18:56:31,448 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-05 18:56:31,648 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-05 18:56:31,664 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/.tmp/rep_barrier/087c2e160e6b401791ad820032eba3a9 2023-04-05 18:56:31,666 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 087c2e160e6b401791ad820032eba3a9 2023-04-05 18:56:31,791 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-05 18:56:31,791 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-05 18:56:31,848 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-05 18:56:31,848 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-05 18:56:31,848 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-05 18:56:32,049 DEBUG [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-05 18:56:32,073 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/.tmp/table/36244fb165844d3cb85689b7032cebb4 2023-04-05 18:56:32,076 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 36244fb165844d3cb85689b7032cebb4 2023-04-05 18:56:32,077 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/.tmp/info/59f18cbf905549afbf33e1becccc40d9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/info/59f18cbf905549afbf33e1becccc40d9 2023-04-05 18:56:32,080 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 59f18cbf905549afbf33e1becccc40d9 2023-04-05 18:56:32,080 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/info/59f18cbf905549afbf33e1becccc40d9, entries=22, sequenceid=93, filesize=7.1 K 2023-04-05 18:56:32,082 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/.tmp/rep_barrier/087c2e160e6b401791ad820032eba3a9 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/rep_barrier/087c2e160e6b401791ad820032eba3a9 2023-04-05 18:56:32,084 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 087c2e160e6b401791ad820032eba3a9 2023-04-05 18:56:32,084 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/rep_barrier/087c2e160e6b401791ad820032eba3a9, entries=6, sequenceid=93, filesize=5.5 K 2023-04-05 18:56:32,085 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/.tmp/table/36244fb165844d3cb85689b7032cebb4 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/table/36244fb165844d3cb85689b7032cebb4 2023-04-05 18:56:32,087 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 36244fb165844d3cb85689b7032cebb4 2023-04-05 18:56:32,087 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/table/36244fb165844d3cb85689b7032cebb4, entries=9, sequenceid=93, filesize=5.5 K 2023-04-05 18:56:32,088 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1243ms, sequenceid=93, compaction requested=false 2023-04-05 18:56:32,094 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-05 18:56:32,094 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-05 18:56:32,095 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-05 18:56:32,095 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-05 18:56:32,095 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-05 18:56:32,249 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33489,1680720816595; all regions closed. 2023-04-05 18:56:32,256 DEBUG [RS:0;jenkins-hbase4:33489] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/oldWALs 2023-04-05 18:56:32,257 INFO [RS:0;jenkins-hbase4:33489] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33489%2C1680720816595.meta:.meta(num 1680720818529) 2023-04-05 18:56:32,260 DEBUG [RS:0;jenkins-hbase4:33489] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/oldWALs 2023-04-05 18:56:32,260 INFO [RS:0;jenkins-hbase4:33489] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33489%2C1680720816595:(num 1680720817947) 2023-04-05 18:56:32,260 DEBUG [RS:0;jenkins-hbase4:33489] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:32,260 INFO [RS:0;jenkins-hbase4:33489] regionserver.LeaseManager(133): Closed leases 2023-04-05 18:56:32,261 INFO [RS:0;jenkins-hbase4:33489] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-05 18:56:32,261 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-05 18:56:32,262 INFO [RS:0;jenkins-hbase4:33489] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33489 2023-04-05 18:56:32,267 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33489,1680720816595 2023-04-05 18:56:32,267 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-05 18:56:32,267 ERROR [Listener at localhost/43705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@442ceb89 rejected from java.util.concurrent.ThreadPoolExecutor@401367d4[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 2571] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 18:56:32,268 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-05 18:56:32,268 ERROR [Listener at localhost/43705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@170d4398 rejected from java.util.concurrent.ThreadPoolExecutor@401367d4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2571] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 18:56:32,269 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33489,1680720816595] 2023-04-05 18:56:32,269 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33489,1680720816595; numProcessing=1 2023-04-05 18:56:32,270 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33489,1680720816595 already deleted, retry=false 2023-04-05 18:56:32,270 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,33489,1680720816595 expired; onlineServers=0 2023-04-05 18:56:32,270 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39855,1680720815401' ***** 2023-04-05 18:56:32,270 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-05 18:56:32,270 DEBUG [M:0;jenkins-hbase4:39855] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6da5e5d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-05 18:56:32,270 INFO [M:0;jenkins-hbase4:39855] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39855,1680720815401 2023-04-05 18:56:32,270 INFO [M:0;jenkins-hbase4:39855] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39855,1680720815401; all regions closed. 2023-04-05 18:56:32,270 DEBUG [M:0;jenkins-hbase4:39855] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 18:56:32,270 DEBUG [M:0;jenkins-hbase4:39855] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-05 18:56:32,271 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-05 18:56:32,271 DEBUG [M:0;jenkins-hbase4:39855] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-05 18:56:32,271 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680720817659] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680720817659,5,FailOnTimeoutGroup] 2023-04-05 18:56:32,271 INFO [M:0;jenkins-hbase4:39855] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-05 18:56:32,271 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680720817660] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680720817660,5,FailOnTimeoutGroup] 2023-04-05 18:56:32,271 INFO [M:0;jenkins-hbase4:39855] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-05 18:56:32,271 INFO [M:0;jenkins-hbase4:39855] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-05 18:56:32,271 DEBUG [M:0;jenkins-hbase4:39855] master.HMaster(1512): Stopping service threads 2023-04-05 18:56:32,271 INFO [M:0;jenkins-hbase4:39855] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-05 18:56:32,272 ERROR [M:0;jenkins-hbase4:39855] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1605349579) connection to localhost/127.0.0.1:38439 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-05 18:56:32,272 INFO [M:0;jenkins-hbase4:39855] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-05 18:56:32,272 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-05 18:56:32,272 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-05 18:56:32,272 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 18:56:32,272 DEBUG [M:0;jenkins-hbase4:39855] zookeeper.ZKUtil(398): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-05 18:56:32,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-05 18:56:32,273 WARN [M:0;jenkins-hbase4:39855] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-05 18:56:32,273 INFO [M:0;jenkins-hbase4:39855] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-05 18:56:32,273 INFO [M:0;jenkins-hbase4:39855] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-05 18:56:32,273 DEBUG [M:0;jenkins-hbase4:39855] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-05 18:56:32,273 INFO [M:0;jenkins-hbase4:39855] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 18:56:32,273 DEBUG [M:0;jenkins-hbase4:39855] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 18:56:32,273 DEBUG [M:0;jenkins-hbase4:39855] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-05 18:56:32,273 DEBUG [M:0;jenkins-hbase4:39855] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 18:56:32,273 INFO [M:0;jenkins-hbase4:39855] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.80 KB heapSize=393.50 KB 2023-04-05 18:56:32,280 INFO [M:0;jenkins-hbase4:39855] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.80 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/aba0cdde7e4148bdaf5bc6c224b278b7 2023-04-05 18:56:32,284 INFO [M:0;jenkins-hbase4:39855] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for aba0cdde7e4148bdaf5bc6c224b278b7 2023-04-05 18:56:32,284 DEBUG [M:0;jenkins-hbase4:39855] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/aba0cdde7e4148bdaf5bc6c224b278b7 as hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/aba0cdde7e4148bdaf5bc6c224b278b7 2023-04-05 18:56:32,287 INFO [M:0;jenkins-hbase4:39855] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for aba0cdde7e4148bdaf5bc6c224b278b7 2023-04-05 18:56:32,287 INFO [M:0;jenkins-hbase4:39855] regionserver.HStore(1080): Added hdfs://localhost:38439/user/jenkins/test-data/a24cff65-b2f8-63db-c2e5-ba16a632ebf4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/aba0cdde7e4148bdaf5bc6c224b278b7, entries=65, sequenceid=578, filesize=24.6 K 2023-04-05 18:56:32,288 INFO [M:0;jenkins-hbase4:39855] regionserver.HRegion(2948): Finished flush of dataSize ~344.80 KB/353075, heapSize ~393.48 KB/402928, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 15ms, sequenceid=578, compaction requested=false 2023-04-05 18:56:32,289 INFO [M:0;jenkins-hbase4:39855] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 18:56:32,289 DEBUG [M:0;jenkins-hbase4:39855] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-05 18:56:32,292 INFO [M:0;jenkins-hbase4:39855] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-05 18:56:32,292 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-05 18:56:32,292 INFO [M:0;jenkins-hbase4:39855] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39855 2023-04-05 18:56:32,294 DEBUG [M:0;jenkins-hbase4:39855] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39855,1680720815401 already deleted, retry=false 2023-04-05 18:56:32,370 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 18:56:32,370 INFO [RS:0;jenkins-hbase4:33489] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33489,1680720816595; zookeeper connection closed. 2023-04-05 18:56:32,370 ERROR [Listener at localhost/43705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6fa87ecb rejected from java.util.concurrent.ThreadPoolExecutor@401367d4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2571] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 18:56:32,371 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6a36d8a2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6a36d8a2 2023-04-05 18:56:32,371 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): regionserver:33489-0x10155ac4ba30001, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 18:56:32,371 ERROR [Listener at localhost/43705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@18d29041 rejected from java.util.concurrent.ThreadPoolExecutor@401367d4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2571] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 18:56:32,371 INFO [Listener at localhost/43705] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-05 18:56:32,470 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 18:56:32,470 INFO [M:0;jenkins-hbase4:39855] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39855,1680720815401; zookeeper connection closed. 2023-04-05 18:56:32,470 ERROR [Listener at localhost/43705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6c100e60 rejected from java.util.concurrent.ThreadPoolExecutor@59558d26[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4302] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 18:56:32,471 DEBUG [Listener at localhost/43705-EventThread] zookeeper.ZKWatcher(600): master:39855-0x10155ac4ba30000, quorum=127.0.0.1:49755, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 18:56:32,471 ERROR [Listener at localhost/43705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5b9774f3 rejected from java.util.concurrent.ThreadPoolExecutor@59558d26[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4302] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 18:56:32,472 WARN [Listener at localhost/43705] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-05 18:56:32,477 INFO [Listener at localhost/43705] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-05 18:56:32,585 WARN [BP-1242217932-172.31.14.131-1680720812184 heartbeating to localhost/127.0.0.1:38439] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-05 18:56:32,585 WARN [BP-1242217932-172.31.14.131-1680720812184 heartbeating to localhost/127.0.0.1:38439] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1242217932-172.31.14.131-1680720812184 (Datanode Uuid adb8a7fe-aea7-4258-9a4f-d23be48afc65) service to localhost/127.0.0.1:38439 2023-04-05 18:56:32,587 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/cluster_e107b28c-27c6-6ca4-18a0-71d30edffa22/dfs/data/data1/current/BP-1242217932-172.31.14.131-1680720812184] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-05 18:56:32,588 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b72effa1-249e-a9f3-719f-6d67c10f7f9a/cluster_e107b28c-27c6-6ca4-18a0-71d30edffa22/dfs/data/data2/current/BP-1242217932-172.31.14.131-1680720812184] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-05 18:56:32,622 INFO [Listener at localhost/43705] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-05 18:56:32,791 INFO [Listener at localhost/43705] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-05 18:56:32,965 INFO [Listener at localhost/43705] hbase.HBaseTestingUtility(1293): Minicluster is down